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ABSTRACT 


This  study  evaluates  the  qua x tty  of  cost  estimates 
produced  by  each  or  four  cost  progress  r.odels — a  random  v.-aik 
model,  the  traditional  learning  curve  model,  a  production  rate 
model  (fixed-variable  model),  and  a  model  incorporating  both 
learning  curve  and  production  rate  effects  (Bemis  production 
rate  adjustment  model) .  Emphasis  is  on  assessing  the  level  of 
bias  associated  with  these  models  and  determining  the 
influence  of  various  factors  on  model  performance.  Findings 
indicate,  on  average,  the  learning  curve  and  Bemis  models 
underestimate  unit  costs,  while  the  random  wai)<  and  fixed- 
variable  models  overestimate  unit  costs.  Different  factors  are 
evaluated  to  determine  their  significance  in  explaining 
variations  in  the  bias  of  unit  cost  predictions  and 
relationships  between  the  significant  variables  and  model  cost 
prediction  bias  are  described.  Findings  indicate  the  Bemis 
model  is  superior  to  the  other  cost  progress  models  because  it 
exhibits  the  least  bias  and  is  not  significantly  influenced 
(in  terms  of  bias)  by  variations  in  the  factors  considered. 
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I. 


INTRODUCTION 


A.  DEFENSE  SPENDING  AND  PUBLIC  INTEREST 

In  recent  years,  spending  for  national  defense  has 
increasingly  been  the  focal  point  of  public  scrutiny.  This 
trend  can  be  attributed  to  two  underlying  causes--the  grov/inq 
federal  budget  deficit,  and  the  problem  of  cost  overruns  in 
the  acquisition  of  major  weapons  systems. 

1.  The  Federal  Budget  Deficit 

Between  fiscal  years  1980  and  1987,  the 
Department  of  Defense  annual  Budget  Authority 
almost  doubled,  from  $143  billion  to  $281 
billion,  with  its  total  exceeding  $1  trillion 
during  that  period.  This  sharp  increase 
contributed  to  rising  deficits  and  aroused 
public  concern  over  the  ways  defense  dollars 
were  being  spent.  [Ref.  l:p.  8] 

The  Congressional  Budget  Office's  "Economic  and  Budget 
Outlook:  Fiscal  Years  1992-1996"  indicates  that  since  1986, 


the  defense  budget  has  been  on  a  downward  path.  Total  budget 
authority*  for  1991  was  down  approximately  four  percent  from 


1990. 

Moreover, 

projections 

for 

1992 

and  1993  indicate 

further 

decreases , 

[Ref.  3:p. 

84] 

As 

a  result  of  these 

Budget  authority  is  "the  authority  granted  to  a  federal 
agency  in  an  appropriations  bill,  to  enter  into  commitments  that 
result  in  immediate  or  future  spending."  Budget  authority  is  not 
necessarily  the  amount  of  money  that  an  agency  or  department  will 
spend  during  a  fiscal  year.  Instead,  it  is  merely  the  upper  limit 
on  the  amount  of  new  spending  commitments  it  can  make.  [Ref.  2:p. 
176] 
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trends,  it  is  more  important  than  ever  that  the  Department  cf 
Defense  (DoD)  manages  irs  dollar  resources  effectively. 

One  key  area  in  which  costs  must  be  managed 
effectively  is  the  development  and  procurement  of  v;eapons 
systems* . 

During  much  of  the  past  three  decades,  constant  dollar 
unit  costs  for  major  defense  systems  have  grown  much 
faster  than  constant  dollar  total  budgets  for  these 
systems.  The  result  has  been  the  purchase  of  smaller 
quantities  of  new  systems,  delayed  modernization  and 
shrinking  capabilities.  [Ref.  l:p.  10] 

Budget  authority  and  outlays  for  research,  development,  test 

and  evaluation  and  procurement  accounted  for  approximately 

forty  percent  of  the  national  defense  budget  in  1989  [Ref. 

4;p.  A-146].  More  effective  and  efficient  utilization  of 

these  funds  could  result  in  significant  savings  within  the 

DoD,  thus  allowing  scarce  resources  to  be  applied  to  other 

important  requirements. 

2.  Cost  Overruns  in  Weapons  Acquisition 

The  second  underlying  cause  of  increased  public 

scrutiny  of  defense  spending  is  cost  overruns.  The  problem  of 

cost  overruns  in  the  weapons  acquisition  process  has  been  a 

major  source  of  consternation  and  embarrassment  for  the  DoD 

for  many  years.  Numerous  researchers  and  presidential 


-  The  term  weapon  system  normally  refers  to  the  major  item  of 
equipment  and  the  subsystems,  logistical  support,  software, 
construction  and  training  needed  to  operate  and  support  it  [Ref. 
l:p.  9].  For  purposes  of  this  thesis,  weapon  system  refers  only  to 
the  major  item  of  equipment. 


2 


corainissions  during  the  past  thirty  years  have  tcnciudea  *:nat 
tens  of  billions  of  dollars  per  year  could  be  savea  cy 
improving  the  acquisition  process  'Ref.  l:p.  32].  "The 

studies  (have)  repeatedly  urged  Congress  and  the  Defense 
Department  to  correct  five  basic  deficiencies: 


1.  Setting  requirements  for  the  most  sophisticated 
systems  available,  often  irrespective  of  cost; 

2 .  Changes  in  program  and  contract  requirements  caused  by 
changes  in  military  user  preferences,  leading  to 
annual  or  more  frequent  changes  in  program  funding 
levels,  initiated  by  Congress  and  DoD  itself; 

3 .  Lack  of  incentives  for  contractors  and  government 
personnel  to  reduce  program  costs; 

4.  Failure  to  develop  sufficient  numbers  of  military  and 
civilian  personnel  with  training  and  experience  in 
business  management  and  in  dealing  with  industrial 
firms  to  oversee  the  development  and  production  of 
enormous,  highly  technical  industrial  programs;  and 

5.  Underestimated  schedules  and  costs  of  major  programs, 
distorting  the  decision-making  process  for  the 
allocation  of  the  national  budget."  [Ref.  l:p.  32] 


While  progress  has  been  made  in  each  of  the  areas 
identified  above,  there  remains  much  room  for  improvement. 
.Major  defense  procurement  programs  have  repeatedly  experienced 
significant  unanticipated  schedule  delays  and  cost  increases. 
More  than  ninety  percent  of  these  programs  exceed  initial  cost 
estimates  and,  in  the  majority  of  cases,  the  average  increase 
in  cost  has  been  more  than  fifty  percent,  excluding  the 
effects  of  quantity  changes  and  inflation.  [Ref.  l:p.  32-331 
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A  cost  overrun  occurs  when  the  actual  cost  of 
program  exceeds  the  estimated  cost.  Cost  overruns  t'/picaily 
occur  when:  fair  initial  cost  estimates  are  made  cut 

subsequent  actual  costs  are  poorly  managed  and  controlled;  or 
actual  costs  are  well-managed  but  initial  cost  estimates  were 
unrealistic,  ;Ref.  5:p.  i] 

There  are  various  reasons  why  initial  cost  estim.ates 
may  be  unrealistic,  particularly  unrealistically  low.  For 
example,  institutional  incentives  may  exist  both  within  DoD 
and  at  government  contractors  to  underestimate  costs  initially 
in  order  to  get  a  program  approved  and  started.  A  second 
possible  reason  is  that  processes,  techniques  or  tools  for 
creating  cost  estimates  may  be  weak  ar.d  provide  misleading 
cost  forecasts.  This  thesis  addresses  a  technical  question 
related  to  the  latter  issue.  In  particular,  this  paper 
examines  the  problem  of  low/unrealistic  cost  estimates  for 
major  weapons  programs  by  analyzing  the  performance  of 
alternative  cost  estimation  models. 

B.  COST  ESTIMATION  MODELS 

Broadly,  cost  estimation  models  fall  into  two  categories. 
First,  Cost  Estimating  Relationships  (CERs)  attempt  to  explain 
or  predict  the  cost  of  a  "standard"  unit  of  an  item  to  be 
manufactured  or  procured  in  terms  of  variables  reflecting 
qualities,  attributes  or  characteristics  of  the  item.  For 
example,  aircraft  costs  may  be  modeled  in  terms  of  speed  or 
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::ay:caa.  ^eccna,  Ccsr  Prcaress  Moaeis  .ittcmot  to  --xviax:':  .r 
:ro.;ioo  onanqes  in  unit  test  :;i  .tens  over  the  ..le 
troauction  or  p  "ocurer.ent  pregran  in  terns  or  onanqcs  -n  tne 
iircun.stancGS  surrounding  prcduction  or  acquisition.  ■'cr 
-'uuanpie,  costs  nay  be  expected  to  depend  on  t.he  nu.noero 
vcauired  and  the  production  rate  and  thus  unit  costs  ray  to 
rcaeled  in  terrs  of  suen  variables.  The  analysis  in  this 
"hGsis  i;ill  locus  on  the  latter  type  cf  rodei,  cost  progress 
rede  Is . 

1.  The  Learning  Curve 

The  rost  ccrnonly  usea  cost  progress  rodei  .s  the 
learning  curve. 

Learning  curves  have  gained  widespread  acceptance  as  a 
tool  for  planning,  analyzing,  explaining  and  predicting 
the  behavior  of  the  unit  cost  of  items  produced  from  a 
repetitive  production  process  'Ref.  5:p.  11. 

.-.it.hough  learning  curves  were  originally  developed  and  applied 
to  predict  cost  and  time  requirements  for  the  construction  cf 
ships  and  aircraft  during  Vvorld  War  II,  they  have  since  been 
ippiied  in  many  other  manufacturing  and  non-m.anuf  actur  ina 
settings.  The  learning  curve  phenomena  was  first  reported  by 
T.  ?.  Wright  in  the  Journal  of  .aeronautical’  Sciences  in  1536. 
Wrignt  observed  that,  as  the  quantity  of  units  manufactured 
doubles,  the  number  of  direct  labor  hours/cost  associated  with 
the  production  of  an  individual  unit  decreases  at  a  uniform 
Moreover,  the  uniform  rate  of  learning  is  peculiar  to 


rate . 


the  ruanuf acturing  process  being  observed. 

2.  Alternative  Cost  Progress  .Models 

It  is  generally  ackncwledgea  that  ether  r actors,  .n 
addition  to  cumulative  quantity,  influence  unit  cost  and  that 
the  simple  learning  curve  does  not  provide  a  fully  adequate 
description  of  cost  behavior.  As  a  result,  prior  research  has 
attempted  to  improve  the  simple  learning  curve  model  by 
including  additional  variables.  'Ref.  5:p.  21 

There  are  now  multiple  approaches  and  models  available 
for  estimating  the  costs  of  acquisition  programs.  Two  of  the 
most  commonly  used  cost  progress  model  types  are  the  learning 
curve  and  the  production  rate  adjustment  moael.  Other  model 
types  include  the  plateau  model,  the  Stanford-B  model,  the  De 
Jong  model  and  the  S-model.  These  models  are  differentiated 
by  the  variables  included  and  the  underlying  assumptions. 

C.  PURPOSE 

It  is  unclear  at  present  which  model  type  is  most 
appropriate  for  predicting  costs  under  various  manufacturing 
conditions.  The  purpose  of  this  thesis  is  to  evaluate  the 
quality  of  cost  estimates  produced  by  each  of  four  cost 
progress  models — a  random  walk  model,  a  learning  curve  model, 
a  production  rate  model,  and  a  model  incorporating  both 
learning  curve  and  production  rate  effects.  In  conducting 
this  evaluation,  emphasis  will  be  place  on  assessing  the  level 
of  bias  associated  with  each  of  these  models. 
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D.  RESEARCH  QUESTIONS 


This  thesis  will  address  the  following  researcn  questions: 

1 .  Primary  Research  Question 

What  is  the  bias  exhibited  by  available  cost  progress 
models  when  predicting  the  future  unit  cost  of  weapons  systems 
acquired  through  a  continuing  acquisition  program? 

2 .  Subsidiary  Research  Questions 


a.  Are  the  various  available  cost  progress  models 
comparable  in  terms  of  bias? 

b.  Do  particular  models  result  in  less  biased 
estimates  under  certain  conditions? 

c.  What  are  those  conditions  that  affect  the 
performance  of  the  models? 

d.  Can  guidelines  be  established  for  determining  when 
(under  what  conditions  or  circumstances)  it  is 

most  appropriate  to  use  a  particular  model  type? 


E.  SCOPE,  LIMITATIONS  AND  ASSUMPTIONS 

There  are  a  number  of  different  criteria  which  can  be 
examined  in  order  to  assess  various  aspects  of  model 
performance.  Two  of  these  criteria  which  are  particularly 
important  are  the  accuracy  and  the  level  of  bias  associated 
with  a  particular  model.  Accuracy  refers  to  the  degree  of 
error  in  a  model's  prediction,  without  regard  to  the  direction 
of  the  error.  Bias  refers  to  both  the  direction  and  the 
magnitude  of  error.  It  indicates  whether  predictions  made 
using  a  particular  model  underestimate  or  overestimate  actual 
cost.  The  focus  of  this  thesis  will  be  limited  to  an  analysis 
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of  the  bias  associated  with  the  various  nodeis  tested.  One 
purpose  of  the  study  is  to  either  confirn  or  disconrirn  the 
results  of  an  earlier  simulation  study  by  Moses  : Learning 
Curve  and  Rate  Adjustment  Models;  .An  Investigation  of  Siasi 
and  to  determine  whether  or  not  those  results  hold  when 
testing  real  world  data.  A  second  purpose  is  to  extend  the 
analysis  of  bias  to  a  larger  set  of  models. 

F.  ORGANIZATION  OF  THESIS 

The  remainder  of  this  thesis  is  organized  into  four 
chapters.  Chapter  II  provides  a  review  of  the  literature 
dealing  with  various  cost  progress  models.  Chapter  III 
provides  a  description  of  the  sample,  data  and  measures  used 
to  conduct  the  study.  Chapter  IV  provides  an  analysis  of  the 
results.  Finally,  chapter  V  summarizes  the  research  findings 
and  suggests  directions  for  future  research. 
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II.  REVIEW  OF  PREVIOUS  RESEARCH 


Although  rhe  progress  or  learning  curve  tecnnique  of  cost 
estimation  v/as  discovered  prior  to  World  War  II,  several 
decades  passed  before  statistical  studies  of  this  phenomena 
could  be  readily  conducted.  This  situation  can  be  attributed 
to  two  underlying  problems:  (i)  the  available  data  were 
frequently  too  sparse  to  support  statistical  analyses,  and; 
(2)  the  sheer  volume  of  calculations  and  lack  of  powerful 
computers  meant  that  many  laborious  hours  were  required  to 
perform  operations  which  can  today  be  performed  in  a  few 
minutes.  This  second  problem  had  a  strong  inhibiting  effect 
on  researchers.  As  a  result,  it  was  not  until  the  1950 's  that 
significant  statistical  studies  of  the  learning  curve 
phenomena  began  to  be  undertaken.  [Ref.  6:p.  8] 

This  chapter  reviews  the  literature  dealing  with 
development  of  the  learning  curve  and  alternative  rate 
adjustment  models.  In  particular,  it  summarizes  the  findings 
of  some  of  the  major  studies  which  have  been  conducted  in  an 
effort  to  evaluate  the  relative  performance  of  these  models. 
The  discussion  is  organized  chronologically  into  the  following 
groupings:  research  prior  to  1970,  research  during  the 
seventies,  research  during  the  eighties,  and  research  during 


the  nineties. 


A.  SURVEY  OF  RESEARCH  PRIOR  TO  1970 


1.  Hirsch 

In  1952,  Hirsch  ;Ret.  7]  published  the  results  or  t 
five-year  study  of  a  large  United  States  nachine  builder.  The 
purpose  of  the  study  was  to  examine  the  relationship  oetween 
labor  requirements  and  production  volume.  Lot  size  v/as  used 
as  a  measure  of  the  rate  of  production  based  on  the  existence 
of  stable  production  lot  intervals.  In  addition,  practically 
no  changes  in  management  or  plant  and  equipment  occurred 
during  the  period  of  study.  Based  on  the  results  of  his 
study,  Hirsch  concluded  the  relation  between  direct  labor 
requirements  and  lot  size  was  of  little  consequence  in  the 
machining  and  assembling  processes. 

2 .  Cochran 

In  1960,  E.  B.  Cochran  [Ref.  3]  published  an  article 
in  which  he  conducted  a  careful  examination  of  the  basic  cost 
function — the  learning  curve — by  studying  specific 
manufacturing  conditions  and  parameters  which  relate  them  to 
cost  trends.  The  purpose  of  this  study  was  to  probe  various 
learning  curve  applications  in  an  effort  to  develop  new 
concepts  of  learning  curve  analysis  and  revised  applications. 
Based  on  the  results  of  his  study,  Cochran  proposed 
refinements  to  the  basic  learning  curve  concept.  In 
particular,  he  suggested  that  learning  may  not  necessarily 
occur  exactly  once  per  unit  but  instead,  may  occur  either 
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faster  or  slower.  For  example ,  when  planning  airrrar.e  costs 
for  a  four-engine  aircraft,  it  may  be  appropriate  to  consider 
each  engine  pod  as  a  unit  of  learning  rather  than  simply 
consider  the  aircraft  as  a  unit  of  learning. 

Cochran  provided  guidelines  for  accurately  identifying 
the  unit  of  learning  and  identified  other  factors,  in  addition 
to  worker  learning,  that  would  affect  the  rate  of  cost 
reduction  (e.g.,  tooling,  supervision,  parts  design  and 
shortages) .  Moreover,  he  demonstrated  how  changing  rates  of 
learning  and  task  changes  may  result  in  both  shifts  of  the 
learning  curve  and  non-linear  learning.  One  of  the  most 
significant  findings  presented  by  Cochran  was  that; 

Any  change  in  learning  rate  is  equivalent  to  a  change  in 
the  unit  at  which  standard  cost  is  reached.  And  this  in 
turn  generates  a  major  shift  in  the  entire  level  of  cost 
[ Ref .  8 : p .  319]. 

As  a  result  of  this  finding,  Cochran  concluded  "that 
the  determination  of  learning  rate  is  of  major  significance  in 
forecasting  and  controlling  costs."  [Ref.  8:p.  319] 

Cochran  indicated  that  the  shape  of  the  learning  curve 
can  be  critical  in  the  first  100  units  and  pointed  out  the 
fact  that  there  is  a  wide  range  of  error  in  straight  line 
curves.  Accordingly,  he  suggested  that  an  S-curve  pattern  may 
be  more  appropriate  than  the  usual  linear  learning  curve. 
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3.  Alchian 


In  1963,  Alchian  _Ref.  3]  publisnea  rne  results  or  i 
1949  study  conducted  for  the  RAND  Corporation.  The  purpose  or 
this  study  was  to  examine  the  similarity  cf  airframe 
manufacturing  progress  functions  among  various  airframe 
manufacturers.  Statistical  tests  of  the  similarity  of  the 
functions  among  various  airframe  manufacturers  were  performed 
using  World  War  II  data.  In  addition,  the  reliability  of 
predictions  made  with  these  curves  was  assessed. 

The  results  of  this  study  indicated  t.he  progress 
functions  differed  among  various  airframe  types  and 
manufacturing  facilities  both  in  the  amount  and  rate  of  change 
of  required  direct  labor  per  pound  of  airframe.  Alchian 
suggested  that,  for  practical  purposes,  the  use  of  an  average 
of  individual  progress  functions  may  be  appropriate.  By 
applying  this  procedure  to  22  airframes  produced  at  different 
facilities,  the  average  production  error^  was  found  to  be 


Production  error  was  defined  as  follows: 


Production  Error 


Predicted  Manhours  -  Actual  Manhours 
Actual  Manhours 


Direct  labor  requirements  (manhours)  for  the  first  1000  planes  were 
predicted  for  22  aircraft  model — facility  combinations  using  both 
an  industry  progress  curve  and  an  airframe  type  progress  curve. 
The  percentage  error  resulting  from  the  use  of  each  of  these  curves 
v/as  then  computed  for  each  model — facility  combination  using  the 
equation  described  above.  Next,  the  weighted  average  error  per 
facility  (weighted  by  actual  manhours)  associated  with  the  use  of 
the  industry  and  airframe  progress  curves  was  computed  for  each  of 
the  four  major  aircraft  model  groups  examined:  bombers,  fighters, 
trainers  and  transports.  Based  on  these  figures,  the  weighted 
average  error  per  facility  for  all  facilities  was  computed. 
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approximately  2  5  percent.  This  same  result  v/as  CDtained  :or 
the  entire  output  of  any  particular  airframe  proaucea  in  one 
racilitv.  Specific  curves  fitted  to  the  oast  performance  or 
a  particular  manufacturing  facility  resulted  in  margins  of 
error  of  approximately  20  percent. 

Alchian  examined  alternative  relationships  cetxeen 
direct  labor  per  pound  of  airframe,  cumulative  number  of 
airframes,  time  and  rate  of  production.  "The  results  cast 
doubts  on  any  of  the  alternatives  being  better  fits  than  the 
usual  progress  curve."  'Ref.  9:p.  692] 

B.  SURVEY  OF  RESEARCH  DURING  THE  SEVENTIES 
1.  Linder  and  Hllbour  ^ 

Models  which  relate  costs  to  various  cost-driving 
features  or  parameters  (i.e.,  physical/performance  parameters 
of  the  weapons  system)  typically  result  in  reasonable 
estimates  of  future  recurring  unit  procurement  costs  [Ref. 
10:p.  277].  Nevertheless,  Linder  and  Wilbourn  [Ref.  101 
suggested  that  in  addition  to  these  parameters,  various 
characteristics  of  the  procurement  program  itself  (e.g., 
competitive  versus  sole-source  procurement,  single-year  versus 
multiple-year  buys,  and  low  versus  high  production  rates) 
represent  cost-driving  features  which  should  be  accounted  for 
in  the  cost  estimating  procedure. 
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Linder  and  Wilbourn  investigated  the  effect  of 
production  rate  on  recurring  nissile  unit  procurement  costs. 
In  particular,  they  developed  two  models  to  examine  hov; 
production  rate  influences  the  position  and/or  slope  of  the 
recurring  missile  hardware  cost  improvement  curve.  The  first 
model  formulated  unit  recurring  cost  as  a  function  of  a 
constant  "annual"  production  rate.  The  second  .model 
formulated  unit  recurring  cost  as  a  function  of  a  variable 
annualized  production  rate. 

These  models  were  developed  based  on  an  analysis  of 
the  impact  of  production  rate  changes  on  direct  and  indirect 
costs.  Linder  and  Wilbourn  reasoned  that  higher  production 
rates  would  result  in  lower  fixed  costs  per  unit.  Moreover, 
high  production  rates  were  expected  to  lead  to  a  smaller 
percentage  increase  in  indirect  costs  than  direct  costs.  As 
a  result,  lower  overhead  rates  should  be  applied  to  direct 
costs.  Based  on  the  combined  effects  of  lower  fixed  costs  per 
unit  and  lower  applied  overhead  rates,  Linder  and  Wilbourn 
concluded  that  unit  costs  would  be  reduced  at  higher 
production  rates,  ceteris  paribus.  Moreover,  the  cost 
improvement  curve  associated  with  a  high  production  rate  was 
expected  to  lie  below  a  cost  improvement  curve  associated  with 
a  lower  production  rate. 

In  assessing  the  expected  effect  of  production  rate  on 
the  cost  improvement  curve  slope,  the  researchers  made  the 
following  observations: 
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1.  At  high  producrion  races,  average  fixed  direcc  rcscr  arc: 
reduced  and  a’anable  direct:  coses  conscituce  a  larger 
portion  of  total  direct  costs. 

2.  As  direct  costs  fall,  indirect  costs  are  reduced  at 
slower  rate  such  that  overhead  rates  increase. 

These  two  phenomena  have  opposite  affects  on  the  slope  of 
the  cost  improvement  curve.  The  first  one  will  tend  to 
increase  the  slope  of  the  curve  due  to  the  influences  of 
learning  and  other  related  effects  on  variable  directs  costs 
at  a  high  production  rate.  The  latter  phenomena  will  tend  to 
flatten  the  curve  as  production  increases.  As  a  result,  "the 
net  effect  on  the  cost  improvement  curve  slope  depends  on  the 
relative  amounts  of  direct  and  indirect  costs  per  unit  as  well 
as  the  proportion  of  each  cost  category  which  can  be 
considered  as  fixed  or  variable."  [Ref.  10:p.  280] 

Based  on  the  results  of  their  analyses,  Linder  and 
Wilbourn  reached  the  following  conclusions: 

1.  Ceteris  paribus,  higher  production  rates  result  in  lower 
unit  recurring  costs  at  each  production  quantity. 

2 .  Doubling  the  production  rate  lowers  average  unit  costs  by 
approximately  three  to  seven  percent  for  the  quantities 
examined. 

3.  Changing  the  production  rate  has  only  a  slight  influence 
on  the  slope  of  the  unit  cost  improvement  curve. 

4.  The  effects  discussed  above  are  relatively  insensitive  to 
changes  in  the  models'  parameter  values.  [Ref.  10:p.  300] 
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2.  RAND  Studies 

During  the  i970's,  the  RAND  Corporation  conductea  3 
number  of  studies  which  examined  the  relationship  between 
production  rate  and  airframe  costs.  The  specific  objectives 
and  results  of  two  of  these  studies  are  summarized  below, 
a.  Large,  Hof f mayor  and  Kontrovich 

In  1974,  the  results  of  a  study  by  Large,  Hoffmayer 
and  Kontrovich  [Ref.  11]  were  published.  "The  purpose  of  this 
study  was  to  investigate  the  nature,  magnitude  and  causes  of 
the  influence  of  production  rate  on  unit  cost."  'Ref.  11  :p. 
iii]  Based  on  the  assumption  that  production  rate  and  unit 
cost  vary  inversely,  the  researchers  sought  to  develop  an 
estimating  model  to  express  the  relationship  for  various 
elements  of  cost. 

The  results  of  the  analysis  suggested  the  effect  of 
production  rate  on  manufacturing  labor,  manufacturing 
materials,  tools  and  engineering  could  not  be  predicted  with 
confidence.  In  any  specific  case,  the  effect  depended  on  a 
number  of  factors  including  how  rate  changes  were  achieved, 
the  availability  of  suppliers,  the  local  labor  supply, 
management  policy,  the  timing  of  rate  changes,  plant  capacity, 
and  plant  backlog.  The  only  element  of  cost  which  was  found 
to  clearly  be  a  function  of  production  rate  was  overhead. 

Based  on  their  findings,  the  researchers  concluded 
that  the  influence  of  production  rate  on  aircraft  cost  could 
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not  be  predicted  with  any  degree  of  confidence.  Eacn  case 
should  be  examined  separarely  and  in  detail  ~o  assess  the 
effect  of  rate.  In  addition,  they  suggested  that  in  advanced 
planning  studies,  rate  effects  in  aircraft  production  programs 
can  be  ignored  because  they  are  far  outweighed  by  other 
uncertainties.  .4s  a  result,  they  indicated  that  a  model  that 
does  not  explicitly  consider  race  may  be  preferable  for 
advance  planning  purposes. 

b.  Large,  Campbell  and  Cates 

This  study,  published  in  1976,  attempted  to  derive 
improved  parametric  equations  for  estimating  the  acquisition 
cost  of  aircraft  airframes.  Earlier  RAND  studies  had 
indicated  variations  in  cost  among  different  airframes  were 
best  explained  by  the  quantity  produced  and  aircraft 
characteristics,  i.e.,  airframe  unit  weight  and  maximum  speed. 
Large,  Campbell  and  Cates  [Ref.  12]  were  unable  to  identify 
additional  characteristics  that  would  make  an  estimating  model 
more  flexible  and,  hence,  better  able  to  deal  with 
characteristics  peculiar  to  individual  aircraft.  None  of  the 
independent  variables  considered  significantly  improved  the 
reliability  of  estimates  obtained  using  only  weight,  speed  and 
cumulative  quantity.  As  a  result,  they  suggested  that  future 
research  which  examines  the  influence  of  program 
characteristics  on  program  cost  may  be  more  productive. 
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Smith 


3  . 

The  purpose  of  ohis  study  was  to  develop  and  test 
procedure  to  assess  the  effect  of  production  rate  changes  cn 
the  direct  labor  requirements  for  production  of  additional 
airframes.  Smith  [Ref.  13]  proposed  a  cost  model  to  express 
direct  labor  hours  required  as  a  function  of  cumulative 
production  and  production  rate.  Two  approaches  to  expressing 
the  production  rate  variable  were  examined:  a  lot  average 

delivery  rate,  and  a  lot  average  manufacturing  rate.  Data 
from  three  airframe  production  programs--the  F-4  program,  the 
F-102  program,  and  the  KC-135  program — were  used  to  construct 
data  sets.  These  data  sets  were  then  examined  in  the 

cumulative  production  and  production  rate  cost  model  using 
regression  analysis. 

Production  rate  was  found  to  be  an  important  factor  in 
the  cost  of  airframe  production,  although  its  effect  was 
subordinate  to  that  of  cumulative  production.  The  study 
demonstrated  empirically  that  production  rate  can  be  an 
important  predictor  of  variation  in  unit  direct  labor 
requirements.  In  addition,  the  results  suggested  an  increase 
in  rate  up  to  plant  capacity  can  lead  to  a  decrease  in  unit 
labor  requirements. 


SURVEY  OF  RESEARCH  DURING  THE  EIGHTIES 


C. 

1.  Crouch 

The  purpose  or  tr.rs  stud'/  '•••'as  to  rnvGstiqace  possic^o 
sources  or  bias  :n  the  Sicpes  or  progress  r unctions  -.vnicn  arc 
convent  ionai  iy  estimated  using  the  unit  cost  progre'is 
runction.  Crouch  ;Ref.  14;  asserted  that  given  the  fact  that 
unit  costs  are  a  function  not  only  of  cumulative  output  but 
also  of  the  rate  of  output  per  time  period,  the  conventional 
unit-cost  progress  function  has  omitted  variables.  Under 
these  circumstances,  t.ne  use  of  ordinary  least  squares 
regression  may  introduce  specification  bias  into  the 
estimates.  Crouch  confirmed  this  situation  mathematically . 

When  a  variable  from  the  true  relation  is  omitted,  a  part 
of  its  influence  in  explaining  the  movements  of  the 
dependent  variable  is  captured  by  the  independent 
variables  which  are  included.  When  the  omitted  variable 
is  not  correlated  with  any  of  the  independent  variables, 
the  coefficients  of  the  included  variables  are  not  biased. 
[Ref.  14:p.  42] 

Crouch  conducted  a  pilot  study  to  investigate  the 
existence  of  bias  when  progress  functions  are  estimated  in  the 
conventional  manner.  Unit-cost  data  (on  an  annual  average 
basis)  in  constant  dollars  for  ten  components  of  the  Hawk 
missile  were  used.  The  results  of  this  study  indicated  that 
v/hen  progress  functions  are  estimated  in  the  conventional 
manner,  biased  estimates  of  the  slopes  may  be  obtained  in  a 
significant  number  of  cases.  In  particular,  the  results 
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indicated  that  when  returns  to  scale  are  constant,  tr.e 
estimated  learning  curve  exponent  obtained  rrci?,  one 
conventional  progress  function  will  be  unbiased.  However, 
when  the  returns  to  scale  are  not  constant,  the  estiniatea 
learning  curve  exponent  obtained  from  the  conventional 
progress  function  will  be  biased.  Negative  bias  will  occur 
when  returns  to  scale  are  increasing;  positive  bias  will  occur 
when  returns  to  scale  are  decreasing. 

2 .  Smith 

The  objective  of  this  study  was  to  examine  the  impact 
of  production  rate  changes  on  the  unit  cost  of  weapons 
systems.  Smith  [Ref.  15]  provided  a  summary  of  the 
significant  research  on  the  relationship  between  production 
rate  and  weapon  system  cost.  In  doing  so,  he  analyzed  the 
various  findings  and  conclusions  and  assessed  their 
applicability.  The  results  of  his  research  review  indicated 
that  only  rather  weak  conclusions  could  be  drawn  from  the 
existing  state  of  knowledge.  The  principal  findings  were: 
production  rate  affects  unit  costs  but,  in  most  cases,  not  as 
strongly  as  the  learning  (cumulative  quantity)  effect,  and; 
the  rate  effect  varies  with  the  weapon  system.  Smith 
identified  four  principal  cost-rate  models — Womer  [Ref.  16], 
Washburn  [Ref.  17],  Linder  and  Wilbourn  [Ref.  10],  and  Fazio 
and  Russell  [Ref.  18].  Each  of  these  models  differed  in  the 
concept  of  rate,  the  number  of  parameters  to  estimate,  and  the 
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range  of  applicable  programs.  'Jevertheless ,  none  of  them  v;as 
considered  suitable  for  use  by  cop  level  budget:  planners. 
^  Ref  .  15  :  p .  ii  ] 

Case  studies  were  conducted  using  production  data  for 
six  missile  systems.  These  systems  possessed  a  wide  range  of 
production  characteristics  ranging  from  low  volume,  labor 
intensive  to  high  volume,  highly  automated  production.  The 
results  of  the  empirical  research  supported  the  belief  that 
under  program  stretch-out,  the  most  important  contributor  to 
increased  unit  costs  is  an  increased  overhead  allocation.  In 
addition,  the  idea  that  labor  inefficiency  is  often  a 
relatively  unimportant  factor  in  rate  adjustments  was 
supported.  [Ref.  15:pp.  43-44]  Based  on  the  results  of  the 
case  studies,  Smith  concluded  that: 

1.  A  simple  rate-sensitive  model  which  focuses  on  the  effect 
of  rate  changes  on  overhead  is  appropriate  for  the 
programming  and  budgeting  phases. 

2 .  Long-range  planners  should  disregard  rate  behavior  and 
focus  only  on  military  requirements.  In  long-term 
planning,  production  rates  and  their  effects  are  both 
unpredictable  and  much  less  important  than  other  more 
fundamental  considerations.  [Ref.  I5:p.  46] 

3.  Balut 

Standard  use  of  learning  curve  theory  involves  an 
implicit  assumption  that  overhead  is  100  percent  variable  with 
direct  costs.  However,  plant  overhead  is  actually  comprised 
of  three  components — variable  overhead,  fixed  overhead,  and 
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semi-fixed  overhead.  Variable  overhead  costs  vary  with  rhe 
activity  rate.  They  include  production-related  indirect  costs 
that  are  tied  to  the  number  of  direct  laborers  working  in  the 
plant  and  the  number  of  units  being  produced.  Fixed  overhead 
costs  do  not  vary  with  the  activity  rate  and  are  fixed  in  the 
short-term  (e.g.,  depreciation,  insurance,  rent,  security). 
Semi-fixed  overhead  is  indirect  expenses  that  are  partially 
fixed  and  partially  variable  such  as  utilities.  Semi-fixed 
overhead  costs  are  typically  gathered  and  reported  as  pools; 
consequently,  the  fixed  and  variable  portions  are  not 
discernable.  [Ref.  19:pp.  63-65] 

The  costing  of  alternative  aircraft  procurement 
quantities  and  rates  within  the  Office  of  the  Secretary  of 
Defense  (Program  Analysis  and  Evaluation)  is  a  two  step 
process  which  considers  the  heterogeneous  nature  of  overhead. 
First,  new  average  unit  prices  are  derived  for  each  lot, 
consistent  with  new  lot  quantities,  using  the  learning  curve. 
Then,  in  order  to  correct  for  the  erroneous  underlying 
assumption  in  step  one,  prices  are  adjusted  to  reflect  the 
redistribution  of  fixed  overhead  resulting  from  a  change  in 
the  production  rate.  This  second  step  is  referred  to  as  rate 
adjustment.  [Ref.  19 :p.  65] 

The  objective  of  Balut's  [Ref.  19]  study  was  to 
evaluate  the  rate  adjustment  model  used  by  the  Office  of  the 
Secretary  of  Defense  by  comparing  its  predictions  to  actual 
contractor  performance.  An  improved  version  of  the  model 


derived  using  actual  contractor  data  was  presented.  3aiut 
illustrated  the  use  of  this  improved  model  for  siruations  wnen 
the  contractor  has  only  one  program,  and  for  situations  when 
the  contractor  has  other  ongoing  programs. 

4 .  Bemis 

Unit  costs  for  weapons  systems  have  traditionally  been 
projected  using  the  experience  curve.  This  method  expresses 
the  projected  unit  cost  as  a  function  of  cumulative  quantity 
produced,  regardless  of  the  production  rate.  Prior  research 
into  the  production  rate-cost  relationship  for  weapons  systems 
indicates  that  unit  cost  varies  significantly  as  a  function  of 
production  rate.  These  variations  are,  to  a  great  extent,  due 
to  the  amortization  of  fixed  overhead.  [Ref.  20:pp.  84-85] 

Bemis  [Ref.  20]  proposed  a  method  for  estimating 
rate/cost/quantity  relationships  using  system  specific  cost 
estimates.  Only  unit-fly-away  costs  were  considered.  The 
input  data  for  the  model  was  historical  rate/cost/quantity 
data  for  ongoing  programs,  and  contractor  or  in-house 
estimates  for  new  programs.  An  equation  was  derived  by 
regressing  unit  cost  on  cumulative  quantity  and  production 
rate.  In  most  of  the  cases  analyzed  in  this  study,  a  high 
multiple  correlation  coefficient  (greater  than  0.9)  was 
obtained. 

Bemis  found  that  when  the  production  rate  was  stable, 
the  experience  curve  method  and  the  rate/cost/quantity  method 
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However,  v/hen  the 


generaced  identical  unit  cost  estimates, 
production  rate  was  variable,  lower  unit  costs  were  associatea 
with  higher  production  rates,  and  higher  unit  costs  were 
associated  with  lower  production  rates. 

Bemis  suggested  the  rate/cost/quantity  model  could  be 
an  invaluable  tool  for  approaching  "what  if"  questions  in  the 
planning  and  budgeting  process.  Moreover,  he  suggested  this 
model  offered  users  a  means  for  readily  assessing  the  cost 
effects  of  program  stretchouts,  the  costs  of  maintaining  a 
warm  production  base,  and  the  probable  effects  of  program 
acceleration. 

D.  SURVEY  OP  RESEARCH  DURING  THE  NINETIES 
1.  Boger  and  Liao 

In  an  effort  to  reflect  the  effect  of  production  rate 
on  the  cost  of  weapons  systems,  researchers  have  proposed  a 
variety  of  adjustments  to  weapons  systems  cost  models.  The 
most  popular  approach  has  been  to  augment  the  traditional 
learning  curve  by  adding  a  rate  term.  The  resulting  learning 
curve  is  referred  to  as  a  rate  adjustment  model.  [Ref.  21: p. 
32]  Boger  and  Liao  [Ref.  21]  examined  the  effects  of 
different  rate  measures  and  cost  structures  on  rate  adjustment 
models  and  illustrated  how  alternative  surrogate  production 
rate  measures  might  lead  to  erroneous  conclusions. 

The  effect  of  production  rate  on  unit  cost  stems  from 
economies  of  scale.  As  production  rates  are  increased, 
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facilities  are  utilized  r^ore  fully  and  greater  spec la i  i zar  icn 
or  labor  occurs.  Materials  costs  are  reduced  because  che 
increased  volume  of  materials  purchased  results  in  quantity 
discounts.  Finally,  the  increased  production  volume  allows 
fixed  overhead  charges  to  be  spread  over  a  larger  quantity  of 
output.  Together,  these  underlying  effects  work  to  increase 
efficiency  and  lower  production  costs,  [Ref.  21:p.  83] 

Increases  in  production  rate  are  normally  expected  to 
result  in  lower  unit  costs  due  to  economies  of  scale;  however, 
production  rate  increases  may  also  lead  to  diseconomies  of 
scale.  Such  is  the  case  when  a  plant  is  operating  beyond  its 
efficient  capacity  level.  Under  these  circumstances,  factors 
such  as  over-time  pay,  lack  of  skilled  labor  or  the  need  to 
invest  in  additional  tooling  and/or  facilities  may  lead  to 
inefficiencies  and  increased  unit  costs.  [Ref.  21:pp.  23-24] 
Because  of  the  difficulty  associated  with  measuring 
production  rate,  a  number  of  alternative  surrogate  measures 
have  been  adopted.  The  two  primary  surrogate  production  rate 
measures  are  lot  size  and  annual /monthly  production  quantity. 
Unfortunately,  there  are  weaknesses  associated  with  the  use  of 
each  of  these  measures.  These  weaknesses  are  as  follows: 

1.  Lot  Size:  The  time  required  to  produce  successive, 

comparably-sized  lots  frequently  changes  over  the  life  of 
a  program.  As  a  result,  it  is  unclear  what  is  being 
measured  by  the  lot  size  proxy. 

2.  Production  Ouantitv/Time  Interval:  If  there  is  a  large 
amount  of  work- in-progress  and  the  production  period  is 
long  compared  to  the  observation  period,  units  produced 
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in  the  following  time  period  will  actually  reflect  worK 
performed  in  the  previous  time  period.  This  tan  result 
in  substantial  bias  in  estimation. 

3.  Average  Rate  for  Each  Program:  This  approach  may 

understate  the  effect  of  descriptive  rate  changes.  An 
average  rate  for  each  program  is  usually  used  in  cross- 
sectional  analysis  because  the  production  rate  may  change 
in  a  typical  production  run, 

4.  Cumulative  Quantity:  Cumulative  quantity  is  highly 

correlated  with  each  of  the  surrogate  production  rate 
measures  discussed  above.  As  a  result,  analysts  have 
been  unable  to  separate  statistically  t.he  effect  of 
learning  and  production  rate.  [Ref.  21:pp.  36-87] 


In  order  to  avoid  some  of  the  difficulties  associated 
with  using  these  surrogate  production  rate  measures,  Boger  and 
Liao  recommended  a  ratio  of  these  measures  be  used.  This 
ratio  should  be  keyed  to  a  base  production  rate.  Adoption  of 
this  approach  offers  a  number  of  advantages: 

1.  Using  a  surrogate  production  rate  ratio  tends  to  mitig~te 
the  multicollinearity  problem. 

2.  Using  the  rate  to  which  the  manufacturer  has  tooled  as 
the  base  rate  provides  an  indicator  of  returns  to 
variable  inputs.  Ratios  greater  than  one  indicate 
decreasing  returns  while  ratios  less  than  one  indicate 
increasing  returns.  [Ref.  21 :p.  88] 

In  addition  to  examining  the  problems  associated  with 
the  two  primary  surrogate  rate  measures,  Boger  and  Liao 
examined  the  problem  of  changing  cost  structures.  These 
changes  occur  as  a  result  of  changes  in  the  production  setup. 
Based  on  their  analysis,  the  researchers  concluded  that  rate 
adjustment  models  are  appropriate  only  when  applied  to  data 
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collected  from  plants  which  have  not  undergone  changes  in  cost 
structure . 

2.  Moses 

In  1991,  Moses  'Ref.  5]  published  the  results  of  a 
study  in  which  he  investigated  and  compared  the  forecasting 
bias  for  the  learning  curve  model  and  a  rate  adjustment  model. 
Specific  objectives  of  the  study  were  as  follows:  to 
determine  if  either  the  learning  curve  model  or  rate 
adjustment  model  exhibits  consistent/systematic  bias;  to 
determine  under  what  circumstances  the  two  models  are  biased, 
and;  to  identify  the  nature  of  the  bias  (i.e.,  overestimation 
or  underestimation  of  future  costs) . 

A  simulation  approach  was  used  to  conduct  the 
research.  First,  cost  series  were  generated  under  varying 
simulated  conditions.  Then,  model  parameters  were  estimated 
by  fitting  the  learning  curve  and  rate  adjustment  models  to 
the  cost  series.  Future  costs  were  predicted  using  each  of 
the  models.  These  predicted  future  costs  were  then  compared 
with  the  actual  cost  to  measure  bias.  Finally,  the 
relationship  between  the  level  of  bias  and  the  simulated 
conditions  was  investigated  using  analysis  of  variance. 

The  simulation  was  conducted  by  varying  seven  factors 
which  had  been  found  to  affect  the  magnitude  of  model 
prediction  errors  in  prior  research.  These  seven  factors 
were: 
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1.  Data  History — the  number  of  data  points  available  tc 
estimate  rhe  parameters  for  a  model. 

2.  Variable  Cost  Learning  Rate — the  learning  curve  exponent. 

3.  Fixed  Cost  Burden — the  proportion  of  total  cost  comprisea 
of  fixed  costs. 

4.  Production  Rate  Trend — the  production  trend  during  the 
model  estimation  period  (i.e.,  gradual  growing  trend  or 
level  trend) . 

5.  Production  Rate  Instabilitv/Variance — period-to-period 
fluctuations  in  the  production  rate  perhaps  caused  by 
changes  in  the  demand  for  output  or  the  supply  of  inputs, 
and  annual  budget  uncertainties. 

6.  Cost  Noise  Variance — variability  in  period-to-period 
cost — designed  to  reflect  unsystematic,  unanticipated, 
non-recurring  random  factors  (e.g.,  changes  in  the  cost, 
type  or  availability  of  input  resources,  temporary 
variations  in  the  level  of  efficiency,  and  unplanned 
changes  in  the  production  process) . 

7 .  Future  Production  Level — the  production  rate  planned  for 
the  future  relative  to  past  levels. 


Bias  was  measured  separately  for  each  model  as  follows: 


BIAS  = 


PUC  -  AUC 
AUC 


where 


PUC  =  Predicted  unit  cost  either  the  learning  curve  or 
the  rate  adjustment  model. 

AUC  =  Actual  unit  cost  generated  by  the  cost  function. 


In  conducting  his  analysis,  Moses  found  that  the  rate 
adjustment  model  provided  unbiased  cost  estimates  while  the 
learning  curve  consistently  underestimated  actual  costs.  The 
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following  conclusions  regarding  learning  curve  bias  were 
drawn : 


1.  Learning  curve  bias  stems  from  the  fact  that  a  portion  of 
total  cost  is  fixed.  The  log  linear  relationship  between 
cost  and  quantity  assumed  by  the  learning  curve  does  not 
hold  when  fixed  costs,  which  are  not  subject  to  learning, 
are  present. 

2.  Bias  increases  as  the  proportion  of  total  cost  made  up  of 
fixed  costs  in  reases.  This  relationship  holds  up  to  the 
point  where  fi^ed  costs  account  for  50  percent  of  total 
cost;  further  increases  above  that  level  reduce  bias. 

3 .  The  production  rate  during  the  period  of  model  estimation 
and  the  production  rate  during  the  period  for  which  costs 
are  forecast  both  affect  the  degree  of  bias.  Bias  is 
minimized  when  there  is  a  consistent  production  trend 
during  these  periods.  Bias  is  magnified  when  there  is  a 
shift  in  production  rate  trend. 

4.  The  steeper  the  learning  curve  slope,  the  greater  the 
level  of  bias.  (This  conclusion  is  based  on  the 
assumption  that  the  proportions  of  total  cost  that  are 
fixed  or  variable  remain  relatively  stable.) 

5.  The  greater  the  number  of  observations,  the  higher  degree 
of  bias. 

6.  The  further  into  the  future  predictions  are  made,  the 
greater  the  degree  of  bias. 


E,  SUMMARY 

The  review  of  previous  research  conducted  in  this  chapter, 
while  comprehensive,  is  by  no  means  all-inclusive. 
Nevertheless,  the  studies  discussed  do  provide  a  sound  basis 
for  assessing  the  current  level  of  understanding  with  respect 
to  learning  curves  and  rate  adjustment  models.  The  following 


onclusions  can  be  drawn  concerning  the  research  efforts  to 


ate : 


1.  The  broad  objective  of  most  of  the  studies  dealing  v/ith 
extensions  or  modifications  to  the  learning  curve  has 
been  to  investigate  the  relationship  between  production 
rate  and  cost  to  determine  whether  or  not  consideration 
of  production  rate  leads  to  improved  cost  estimates. 

2.  Findings  concerning  the  importance  of  considering  the 
effect  of  production  rate  on  unit  costs  when  predicting 
weapons  systems  costs  have  been  inconclusive.  Some 
studies  have  found  production  rate  to  be  an  important 
predictor  of  cost  variation  while  other  studies  found 
production  rate  to  be  of  little  or  no  significance. 

2.  Production  rate  affects  unit  cost  but,  in  most  cases,  not 
as  strongly  as  the  learning  (cumulative  quantity)  effect. 

4.  The  production  rate  effect  varies  with  the  weapon  system 
and  the  specific  cost  elements. 

5.  The  influence  of  production  rate  on  unit  cost  depends  on 
the  relative  amounts  of  direct  and  indirect  costs  per 
unit  and  the  proportion  of  each  category  that  can  be 
considered  as  fixed  or  variable. 

6.  Variations  in  unit  cost  in  response  to  production  rate 
changes  stem  from  economies  of  scale.  These  variations 
are,  to  a  great  extent,  due  to  the  amortization  of  fixed 
overhead. 

7.  Lower  unit  costs  are  associated  with  high  production 
rates  and  higher  unit  costs  are  associated  with  lower 
production  rates. 

8.  The  rate  adjustment  model,  as  described  by  Moses  (1991) , 
provides  unbiased  cost  estimates.  Conversely,  the 
learning  curve  consistently  underestimates  actual  costs. 

9.  Learning  curve  bias  stems  from  the  fact  that  a  portion  of 
total  cost  is  fixed. 

10.  Learning  curve  bias  is  affected  by  the  proportion  of 
total  cost  made  up  of  fixed  costs,  the  production  rate 
during  both  the  period  of  model  estimation  and  the  period 
for  which  costs  are  forecast,  the  slope  of  the  learning 
curve,  the  number  of  observations,  and  the  time  horizon 
for  which  predictions  are  being  made. 
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Findings  by  Moses  [Ref.  5]  concerning  the  oias  associarea 
v/ith  the  traditional  learning  curve  and  the  rate  adjustment: 
model  are  noteworthy.  Nevertheless,  their  significance  is 
somewhat  tempered  by  the  use  of  simulated  data.  The  remainder 
of  this  thesis  will  focus  on  confirming  or  disconf irming  the 
results  of  the  Moses  'Ref.  5]  study  and  will  extend  the 
research  to  include  two  additional  models--a  random  walk  model 
and  a  model  incorporating  production  rate  effects. 
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III.  SAMPLE,  DATA  AND  VARIABLES 


The  purpose  of  rhas  chapter  is  to  provide  an  cverviev.  _ : 
the  methodology  used  to  conduct  this  study.  -he  cnaptar  ..’ill 
begin  with  a  description  of  the  population  iron  v;hich  the 
sample  was  drawn,  the  criteria  for  inclusion  of  a  weapon 
system/procurement  program  in  the  sample  and  the  specific  raw 
data  collected  for  each  weapon  system.  This  will  be  followed 
by  a  description  of  the  procedure  used  to  "expand"  the  sample, 
hext,  the  four  cost  progress  models  that  were  included  in  the 
study  will  be  introduced  along  with  the  procedure  used  for 
predicting  costs  with  these  models.  Finally,  the  procedure 
used  to  measure  bias  in  the  study  will  be  explained  and  the 
"demographic  variables"  and  the  "condition  variables"  that 
were  selected  as  likely  candidates  for  being  significant 
explainers  of  bias  will  be  defined. 

A.  SELECTION  OF  THE  SAMPLE 
1.  Data  Sources 

Data  used  in  conducting  the  study  were  obtained  rrom 
two  sources;  the  U.S.  Military  Aircraft  Cost  Handbook  [Ref. 
22],  and;  the  U.S.  Missile  Cost  Handbook  [Ref.  23].  Data 
contained  in  these  handbooks  were  based  on  historical  Service 
cost  data  and  reflected  the  annual  total  obligational 
authority  (TOA)  flyaway  costs  for  the  included  programs. 


Aircraft  data  fell  inro  five  major  categories:  attacK 
iircrnrt,  fighter  aircrart,  bonoers,  attack  .cei  icopcers ,  ana 
carrel  aircraft.  Missile  data  fell  into  four  mam  categories: 
air-ro-air  m.issiies,  air-ro-surf ace  missiles,  surface-to-air 
missiles,  and  surface-to-surface  missiles.  Ail  cost  data 
contained  in  these  handbooks  was  normalized  to  a  constant 
fiscal  year  base  {FY-81).  Consequently,  the  consistency  of 
t.his  data  was  ensured  for  all  aircraft  and  missile  programs. 

Weapons  systems  contained  in  the  aircraft  handbook 
were  I’.S.  Navy,  Air  Force,  Marine  Corps  and  .Army  aircraft. 
These  aircraft  were  combat-oriented  and  were  in  the  active 
U.S.  inventory  during  the  FY  1960-1980  period.  Trainers, 
reconnaissance  or  electronic  warfare  variants  and  those 
aircraft  produced  for  foreign  military  sales  were  not  included 
in  the  handbook. 

The  missile  handbook  contained  cost  information  for 
T.3.  .Air  Force,  Navy,  and  Army  missile  programs  during  the 
period  FY  1961-1983. 

Costs  reflected  in  these  handbooks  were  TOA  dollars — 
the  amounts  budgeted  in  a  specific  fiscal  year.  TOA  dollars 
do  not  reflect  actual  expenditures  in  any  given  fiscal  year. 
N'evertheless ,  TOA  dollars  do  provide  an  excellent  proxy  for 
actual  dollar  expenditures  because  it  is  customary  within  the 
DoD  to  ensure  that  expenditures  match  total  obligational 
authority  prior  to  the  lapsing  of  an  appropriation.  As  a 


result,  differences  between  TOA  and  actual  expenditures  are 
norraaily  very  small. 

2.  Selection  Process  and  criteria 

Selection  of  aircraft  and  missile  programs  for 
inclusion  in  this  study  was  based  on  three  criteria:  the 

number  of  plot  points  or  fiscal  years,  the  rvailability  of 
airframe  cost  data  and  the  completeness  of  the  data.  Only 
programs  for  which  five  or  more  fiscal  years  worth  of  data  v/as 
available  were  included  in  the  study.  This  criterion  was 
required  to  ensure  the  minimum  amount  of  data  necessary  for 
statist ically  fitting  the  cost  progress  models.  The  latter 
two  criteria  were  established  to  ensure  that  meaningful 
analyses  could  be  conducted.  As  a  result,  only  aircraft 
programs  which  had  complete  quantity  and  airframe  cost  data 
were  included  in  the  study.  In  addition,  only  missile 
programs  for  which  complete  quantity,  and  guidance  and 
control/airframe  costs  were  available  were  included  in  the 
study. 

3 .  Programs  Selected  for  study 

Based  on  the  three  criteria  described  above,  forty-six 
weapons  procurement  programs  (fourteen  missile  programs  and 
thirty-two  aircraft  programs)  were  selected  for  inclusion  in 
this  study.  The  specific  aircraft  and  missile  programs  that 
comprised  the  sample  are  included  in  Tables  1  and  2 
respectively. 
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FABLE  1 


AIRCRAFT 

PROCUREMENT 

PROGRAMS 

INCLUDED  IN 

THE  STUDY 

1. 

A-3A/B 

12. 

AH-IS 

23  . 

F-16A 

2  . 

A-4C 

13  . 

AV-8A 

24  . 

F-IOIA/B/C 

3  . 

A-4E 

14  . 

B-47B/E 

25. 

F-102A 

4  . 

A-4A/B 

15. 

F-3A/B/C 

26. 

F-105D 

5  . 

A-4E/F 

16. 

F-4A 

27  . 

F-105B/D 

6 . 

A-6A 

17. 

F-4B 

28. 

F-lllF 

7  . 

A-6E 

18. 

F-4E 

29. 

P-2H 

8  . 

A-7D 

19. 

F-4J 

30. 

P-3A 

9  . 

A-7E 

20. 

F-4A/B 

31. 

P-3C 

10. 

A-37B 

21. 

F-8D/E 

32  . 

S-3A 

11. 

AH-IG 

22  . 

F-14A 
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TABLE  2 


MISSILE  PROCUREMENT  PROGRAMS  INCLUDED  IN  THE  STUDY 


PHOENIX  (AIM-54A) 

3. 

STANDARD  ER 

(RIM-67B/ 

B-l/C-1) 

2  . 

SIDEWINDER  (AIM-9D/G) 

9. 

STANDARD  ER 

(RIM-67A) 

3  . 

SIDEWINDER  (AIM-9H) 

10. 

STANDARD  MR 

(RIM-66A) 

4  . 

SIDEWINDER  (AIM-9H/L) 

11. 

STANDARD  MR 

(RIM-66B) 

5 . 

SPARROW  (AIM-7E) 

12. 

TALOS  (RIM-8E) 

6  . 

SPARROW  ( AIM-7 F) 

13. 

TARTAR  (RIM- 

24B) 

7  . 

SHRIKE  (AGM-45A) 

14. 

TERRIER  (RIM 

-2E) 

TABLE  2 
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B.  COMPILATION  OF  PROGRAM  DATA 


Once  the  sample  of  aircraft  and  missile  procuremenr 
programs  had  been  selected,  the  following  data  were  obtained 
for  each  program  in  the  sample: 


1.  Program  Name — name  of  the  weapon  system. 

2.  Manufacturer — name  of  the  prime  contractor. 

3.  Military  Service — identified  which  Service  branch (es) 
procured  the  weapon  system. 

4.  Program  Type — identified  the  nature  of  each  weapon 
system,  i.e.,  aircraft  or  missile. 

5.  Mission — identified  each  weapon  system  according  to  its 

primary  mission.  Aircraft  program  types  were: 

fighter /attack,  fighter,  attack,  bomber,  patrol,  and  attack 
helicopter.  Missile  program  types  were:  air-to-air,  air-to- 
surface,  surface-to-air,  and  surface-to-surface. 

6.  Modification — identified  whether  a  particular  weapon 
system  was  an  entirely  new  design  or  a  modification  of  an 
existing  design. 

7.  Combined  Program — identified  whether  a  procurement 
program  included  several  versions  of  a  given  weapon  system 
or  only  one  version. 

8.  Fiscal  Year — identified  the  fiscal  years  during  which 
quantities  of  a  particular  weapon  system  were  procured. 

9.  Quantity — identified  the  number  of  units  of  each  weapon 
system  procured  in  a  given  fiscal  year. 

10.  Aircraft  Airframe  Cost/Missile  Guidance  and  Control 
Airframe  Cost 
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EXPANSION  OF  THE  SAMPLE 


Following  the  compilation  of  data  described  in  che 
previous  section,  the  original  data  sample  of  forty-six 
programs  was  partitioned  into  121  cost  series.  This 
"expansion  of  the  sample"  was  accomplished  by  dividing  eacn 
program  cost  series  into  individual,  consecutive  year-to-date 
cost  series.  For  example,  if  a  particular  procurement  program 
had  cost  data  available  for  seven  fiscal  years,  e.g.,  FY  1970- 
197  6,  this  single  cost  series  could  be  expanded  into  four 
separate  cost  series  as  shown  below. 


Cost  Series  #  1: 
Cost  Series  #  2: 
Cost  Series  #  3; 
Cost  Series  #  4: 


FY  1970-1973  (used  to  predict  1974  cost) 
FY  1970-197-,  (used  to  predict  1975  cost) 
FY  1970-1975  (used  to  predict  1976  cost) 
FY  1970-1976  (used  to  predict  1977  cost) 


The  initial  cost  series  for  each  weapon  system  included  in  the 
sample  was  comprised  of  data  from  the  first  four  fiscal  years 
of  that  particular  program  (the  minimum  number  of  years  needed 
to  estimate  the  cost  models) .  Each  subsequent  cost  series  for 
a  given  program  was  then  created  by  additionally  including  the 
data  point  for  the  next  fiscal  year  in  the  existing  cost 
series.  By  partitioning  the  sample  size  in  the  study  in  this 
manner,  it  was  possible  to  simulate  actual  usage  of  the 
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various  cost  progress  models  over  time  and  evaluate  their 
performance  under  varying  data  availability  conditions. 

D.  SELECTION  OF  COST  PROGRESS  MODELS 

Previous  research  has  demonstrated  that  three  factors  are 
particularly  useful  in  predicting  the  future  costs  of  'veapons 
systems:  past  cost,  cumulative  quantity,  and  production  rate. 
Numerous  models  have  been  introduced  in  an  effort  to  improve 
the  quality  of  cost  estimates  over  those  oh'tained  using  the 
traditional  learning  curve  model.  These  models  are 
differentiated  by  the  explanatory  variables  included  and  the 
underlying  assumptions  with  regard  to  the  relative  importance 
of  past  cost,  cumulative  quantity  and  production  rate  in 
predicting  future  costs. 

In  addition,  these  models  may  be  differentiated  by  the 
period  or  length  of  time  over  which  data  is  observed  and  used 
in  creating  a  forecast.  Some  models  assume  that  future  cost 
depends  only  on  the  most  recent  cost,  quantity  and/or 
production  rate  levels.  However,  other  models  assume  that 
cost,  quantity  or  production  rate  levels  from  early  in  a 
program's  life  are  also  significant  and,  as  a  result, 
specifically  consider  data  covering  the  full  production  life. 
Regardless  of  which  approach  is  used,  there  are  botu 
advantages  and  disadvantages  associated  with  the  use  of  data 
comprised  of  only  recent  observations  or  data  comprised  of 
both  recent  and  earlier  observations. 


■-’SG  of  recent  observations  nay  increase  the  relevance  of  noaei. 
results  in  light  of  the  current  situation  but  sacrifices  any 
information  reflected  in  earlier,  historical  data.  In 
addition,  models  based  only  on  a  few  recent  observations  are 
more  susceptible  to  random  noise.  Conversely,  use  of  older 
observations  may  reduce  the  relevance  of  results  or  lead  to 
results  that  are  not  representative  of  the  current  situation. 
However,  the  use  of  additional  data  reduces  the  impact  of 
random  variance  in  recent  observations. 

Finally,  in  addition  to  the  two  factors  discussed  above, 
cost  progress  models  may  be  differentiated  by  the  form  of  the 
assumed  relationship  between  the  dependant  variable — cost — and 
the  potential  explanatory  variables — past  cost,  cumulative 
quantity,  and  production  rate.  Relationships  may  be  linear, 
log  linear  or  some  other  form. 

Four  alternative  models  were  selected  for  inclusion  in 
this  study:  a  random  walk  model,  the  traditional  learning 
curve  model,  a  model  which  expresses  unit  cost  as  a  function 
of  past  cost  and  production  rate  (the  fixed-variable  model) 
and  the  common  rate  adjustment  model. 

1.  Random  Walk  Model 

The  random  walk  model  assumes  that  future  cost  is  a 
function  of  past  cost;  however,  only  the  most  recent  cost  is 
relevant.  Any  deviation  from  predicted  cost  is  considered 
random  deviation. 
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The  model  is  expressed  as  follows: 


—  u 


'C. 


'.vhere 

:’C.  =  Predicted  unit  cose  in  time  period  c. 

I’C.  .  -  =  Actual  unit  cost  in  time  period  t  -  1. 
t  =  Time  period. 

This  model  was  selected  for  inclusion  in  the  study  for  two 
reasons.  First,  it  exemplifies  the  cost  estimation  method 
used  by  budget  programmers  when  there  is  only  very  limited 
historical  data.  Under  these  circumstances,  future  cost 
projections  are  often  based  on  actual  costs  in  the  previous 
period.  Second,  the  random  walk  model  is  the  most  basic  and 
naive  cost  estimation  model  and,  as  such,  provides  a  useful 
benchmark  for  evaluating  the  performance  of  other  more 
sophisticated  models, 

2 .  Traditional  Learning  Curve  Model 

The  second  model  selected  for  inclusion  in  the  study 
v;as  the  traditional  learning  curve.  This  model  assumes  that 
future  cost  is  a  function  of  both  past  cost  and  cumulative 
quantity  produced.  Moreover,  all  historical  cost  data  is 
considered  to  be  relevant. 
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The  traditional  learning  curve  model  assumes  a  log  linear 


relationship  between  cost  and  cumulative  quantity  ana  :s 
expressed  as  follows: 

I'C.  =  aCf 


where 

1/Cj.  =  Incremental  unit  cost  of  item  an  quanzizy  Q. 

Qf.  =  Cumulative  quantity  produced  as  of  period  t. 

a  =  Theoretical  first  unit  cost. 

t  =  . £  =  Learning  curve  exponent . 

log  2 

r  =  Learning  rate. 

The  traditional  learning  curve  model  was  selected  for 
inclusion  in  the  study  because  it  is  the  most  widely 
researched  cost  progress  model  and  represents  the  foundation 
on  which  other  cost  progress  model  variants  are  based. 

3 .  Fixed-Variable  Model 

The  third  model  selected  for  inclusion  in  the  study 
was  the  fixed-variable  model.  This  model  specifically 
addresses  the  relationship  between  total  unit  cost  and  unit 
variable  costs  and  unit  fixed  cost.  Variable  cost  per  unit 
remains  constant;  however,  fixed  cost  per  unit  varies 
depending  on  the  production  volume  because  total  fixed  cost  is 
allocated  by  spreading  it  over  the  total  volume  of  output. 


42 


he  fixed-variable  model  is  expressed  as  follows: 


UC. 


a 


where 

L/C-  =  Unit  cost  in  period  t. 
a  =  Variable  cost  per  unit, 
b  =  Standard  fixed  cost  per  unit. 

PR..  =  Production  rate  =  0^  ^  O^vc' 

-  Production  quantity  in  period  t. 

=  Average  production  quantity  per  period. 

This  model  was  included  in  the  study  because  it  explicitly 
considers  the  impact  of  production  rate  on  unit  cost  through 
the  allocation  of  fixed  overhead.  In  contrast  to  the 
traditional  learning  curve  model  (which  includes  cumulative 
quantity  but  not  production  rate  as  an  explanatory  variable) , 
the  f ixed-variable  model  includes  production  rate  but  ignores 
cumulative  quantity  as  an  explanatory  variable. 

4.  Bemis  Production  Rate  Adjustment  Model 

The  final  model  included  in  the  study  is  the  most 
widely  used  rate  adjustment  model.  This  model,  popularized  by 
Bemis  [Ref.  16],  was  developed  by  augmenting  the  traditional 
learning  curve  model  with  a  production  rate  term. 
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,he  model  is  expressed  as  follows: 


where 

i'C.  =  Predicted  iinic  cost  at  quantity  Q  and  prcducticn  race 
per  perioa  R. 

Q,  =  Cumulative  quantity  produced  as  of  period  c. 

=  Production  race  in  period  t. 
a  =  Theoretical  first  unit  cost, 
b  -  Learning  curve  exponent, 
c  =  Production  rate  exponent. 

The  Bemis  production  rate  adjustment  model  was  included  in  the 
study  because  it  considers  both  cumulative  quantity  and 
production  rate  (in  addition  to  past  cost).  Hence,  it  is  the 
most  comprehensive  of  the  four  models  in  the  study. 

E.  ANALYSIS  OF  BIAS 

1.  Unit  Cost  Prediction 

In  order  to  assess  the  bias  exhibited  by  the  random 
walk  model,  the  traditional  learning  curve  model,  the  fixed- 
variable  model,  and  the  Bemis  model,  predicted  unit  costs  were 
estimated  by  applying  each  of  the  four  models,  in  turn,  to  the 
actual  cost  series.  The  following  paragraphs  describe  how 
this  procedure  was  accomplished. 

The  random  walk  model  assumes  that  unit  cost  in  the 
next  period  is  the  same  as  unit  cost  in  the  current  period. 
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Consequently,  the  predicted  unit  cost  for  any  period  (t)  -./as 
simply  the  actual  cost  from  the  preceding  period  (t-l). 

Predicted  unit  costs  for  the  remaining  three  models 
were  derived  in  the  following  manner.  First,  each  of  the 
models  was  separately  fit  to  the  initial  cost  series 
(comprised  of  the  first  four  fiscal  years'  data)  for  each 
weapon  system  to  derive  the  models'  parameters.  Then,  the 
appropriate  data  values  for  cumulative  quantity  and/or 
production  rate  from  period  five  were  input  into  each  of  the 
models  to  obtain  estimated  unit  costs  for  period  five.  These 
two  steps  were  then  repeated  for  each  remaining  cost  series 
for  the  various  weapons  systems  until  predicted  costs  had  been 
derived  for  every  fiscal  year  for  which  actual  cost  data  was 
available  (e.g.,  models  were  next  estimated  on  five  years 
worth  of  data,  then  used  to  predict  the  cost  for  year  six) . 
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2.  Bias  Measurement-The  Dependant  variable 

Once  predicted  unit  costs  haa  been  computed  rcr  u-icn 
cost  series  with  each  nodel,  a  measure  of  bias  '.vas  detemnea 
for  each  prediction  as  follows: 


•DTjr  _ 


AU'. 


v/here 


BIAS  -  Percentage  difference  between  credicted  unit  cost 
and  actual  unit  cost. 

PUC  =  Predicted  unit  cost  from  the  car ticular  c.cdel  of 
interest . 

AUC  =  Actual  unit  cast  obtained  from  original  unit  cost 
data . 


Positive  BIAS  values  indicate  a  model  has  overestimated  actual 
future  cost;  negative  BIAS  values  indicate  a  model  has 
underestimated  actual  future  cost.  BIAS  values  of  zero 
indicate  the  predicted  cost  and  actual  future  cost  are 
identical  and  the  associated  model  is  unbiased. 

As  in  the  earlier  study  conducted  by  Moses  [Ref.  5], 
BIAS  represented  the  dependent  variable  in  the  statistical 
analysis.  The  basic  objective  of  the  study  was  to  determine 
what  factors  or  conditions  are  useful  in  explaining  variance 
in  BIAS. 
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The  measures  of  bias  and  associated  labels  for  each  of  the 
:our  .-.ode is  v.-ere  as  follows: 


Bias 

for  t.he 

Random  Walk  Model: 

6IASRW 

Bias 

for  the 

Learning  Curve 

Mode  1 : 

BIASLC 

Bias 

for  the 

F ixed-Variable 

Model : 

BIASFV 

Bias 

for  the 

Semis  Model: 

BIASBE 

ven 

121  cost 

series  (from 

the  46 

procurement 

there  were  121  separate  measures  of  bias  for  each  model. 

3.  Explanatory  Variable  Selection 

Model  performance  in  prediction  (i.e.,  the  degree  and 
direction  of  bias)  depends  on  the  circumstances  in  which  the 
model  IS  used.  Two  broad  categories  of  factors  which  might 
influence  model  performance  and,  hence,  be  useful  in 
explaining  bias,  were  identified  in  the  study.  The  first 
category  consisted  of  "demographic  variables"  and  the  second 
category  consisted  of  "condition  variables".  Together,  these 
tv/o  groups  of  variables  were  the  independent  variables  in  the 
statistical  analysis.  Table  3  summarizes  the  independent 
variables  and  their  corresponding  labels. 
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TABLE  3 


INDEPENDENT  VARIABLES 


Demographic  I'ariables 
Program  Type 
Modification 
Military  Service 
Mission 


Labels 


TYPE 

MOD 

MILSERV 

MISSION 


Condition  Variables 
Burden 

Cost  Variance 
Learning  Rate 
Production  Rate  Variance 
Future  Production  Level 
Past  Production  Trend 
Plot  Points 


BURDEN 

CVAR 

LRATE 

RATEVAR 

FUTUPROD 
BEGTREND , 
PLOTPNTS 


ENDTREND 
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a.  Demographic  Variables 


The  demographic  variables  describe  characteristics 
of  the  weapons  procurement  program.  Four  demographic 
variables  were  considered  in  the  study: 

1 .  Program  Type 

2.  Modification 

3.  Military  Service 

4.  Mission 

These  particular  demographic  variables  were 
selected  for  investigation  because  they  represent  readily 
apparent  characteristics  of  the  various  programs  which  may 
affect  model  bias. 

b.  Condition  Variables 

In  addition  to  the  four  demographic  variables, 
seven  condition  variables  were  considered  in  the  study: 

1 .  Burden 

2 .  Cost  Variance 

3 .  Learning  Rate 

4 .  Production  Rate  Variance 

5 .  Future  Production  Level 

6.  Past  Production  Trend 


7.  Plot  Points 


The  label  "condition  variables"  is  used  here  because  eacr.  ~r 
cnese  variables  in  soirie  cianner  indicares  somerhing  iccuc 
conditions  thar  existed  during  the  v/eapon  system  prccuremenr. 
These  are  analogous  independent  variables  (factors)  t.o  chose 
examined  by  Moses  TRef.  5].  The  following  paragraphs  describe 
each  of  these  variables  along  with  the  underlying  rationale 
for  their  inclusion  in  the  study. 
flj  Burden 

Burden  (BURDEN)  indicates  the  percentage  of 
total  unit  cost  made  up  of  fixed  cosr.  Burden  is  measured  as 
follows : 

BURDEN  =  — ^ 
a  +  D 


v/here 

a  =  Variable  cost  per  unit  {constant)  . 
b  =  Standard  fixed  cost  per  unit. 

Note:  a  and  b  were  estimated  parameters  from  the  fixed- 
variable  model. 

Burden  was  included  in  the  study  because  past  research  (Linder 
and  VJilbourn  [Ref.  10],  Moses  [Ref.  5]  and  Smunt  [Ref.  24]) 
has  shown  that  burden  directly  influences  the  impact  of 
changes  in  production  rate  on  unit  cost.  As  production  rate 
increases,  the  cumulative  quantity  produced  during  a  period 
increases  and  the  variable  cost  per  unit  decreases  due  to  the 
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ir.cidence  of  learning.  In  addition,  as  production  rate  in  a 
period  increases,  fixed  cost  per  unit  is  reaucea  as  oooa. 
fixed  cost  is  spread  over  a  larger  production  output.  The 
irpact  of  production  rate  increases  as  the  proportion  of  total 
cost  made  up  of  fixed  costs  increases.  .as  a  result,  the 
relative  bias  of  the  various  models  may  depend  on  t.ne  level  ci 
burden , 

(2)  Cost  Variance 

Cost  variance  (CVAR)  indicates  the  amount  of 
unsystematic  variation  in  unit  cost  that  may  result  from 
unanticipated,  non-recurring,  random  factors.  Examples 
include  changes  in  the  cost,  type  or  availability  of  input 
resources,  temporary  fluctuations  in  efficiency,  and  unplanned 
changes  in  the  production  process. 

By  assessing  the  amount  of  unsystematic 
variation  associated  with  various  weapons  procurement 
programs,  it  may  be  possible  to  determine  whether  there  is  a 
relationship  between  cost  stability  and  bias.  For  this 
reason,  cost  variance  was  examined  in  the  study. 
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Cost  variance  was  measured  as  follows: 


'.vhere 

Cost  vax lance . 

Unit  cost  for  a  given  production  period. 

Average  unit  cost  for  all  production  periods  to  date. 
Total  number  of  production  periods. 

(3)  Learning  Rate 

Learning  rate  (LRATE)  measures  the  decrease  in 
per  unit  cost  that  occurs  as  the  quantity  of  units 
manufactured  doubles.  It  is  affected  by  the  type  of 
production  process  and  the  complexity  of  the  product  design. 
Smunt  [Ref.  24]  found  that  the  degree  of  learning  in  the 
underlying  production  process  determines  the  improvement  in 
prediction  accuracy  that  results  from  including  a  learning 
parameter  in  a  model.  Learning  rate  was  examined  in  the  study 
to  determine  the  nature  of  the  relationship  between  learning 
rate  and  bias  for  the  various  models. 
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laarning  rate  vas  reasurea  as  fDllov;s: 


■..•here 


r  =  Learning  race  excressea  as  a  percencage. 
b  =  Learning  carve  expcnent  as  eszictaced 

by  cha  cradicicnal  learning  carve  model . 


(4)  Production  Rate  Variance 

Production  race  variance  (RATEVAR)  reflects  the 
severity  of  period-to-period  fluctuations  in  production  rate. 
These  fluctuations  inay  result  from  either  changes  in  demand 
for  a  particular  weapon  system  or  changes  in  the  cost  or 
availability  of  production  inputs.  Production  rate  variance 
v;as  measured  as  follows; 

!^g  ~  ^AVGi 

RATEVAR  =  - - 

n 


'■■/here 


Rate  variance. 

C'Uancity  of  amts  produced  in  the  carre.ac  pence. 
Average  quantity  of  units  produced  for  all 
periods  to  date. 

Total  number  of  oeriods  to  date. 


S3 


5)  Future  Production  Level 

Future  prcducrion  level  'FUTUPROD)  .r.aicures 
•'.•herher  the  level  of  production  in  the  next  period  ithe  perioa 
for  '.vhich  cost  is  to  be  forecast  using  the  model)  is  high  or 
low  relative  to  the  current  period.  Future  unit  costs  are 
predicted  using  cost  progress  models  fit  to  past  production 
data.  As  a  result,  the  accuracy  of  the  various  models  nay  be 
affected  if  the  production  level  in  the  period  for  v/hich  costs 
are  being  estimated  varies  significantly  from  the  production 
levels  that  existed  during  previous  periods.  Future 
production  level  was  examined  in  the  study  to  determine 
whether  production  growth  and  production  cutbacks  affect  the 
tendencies  of  the  various  models  to  over/underestimate  unit 
costs . 

Future  production  level  was  measured  as 

follows : 


FUTUPRCD 


loo 


v;here 


FUTUPROD  =  Future  produccion  level. 

Of,  =  Production  level  for  the  next  period  for  which 

costs  are  being  forecast . 

0i  =  Production  level  for  the  last  {most  recent) 

period. 
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cJ  Past  Prcducticr.  Trend 

?asr  prcauccion  rrena  indicares  ~he  pacrarr.  a: 
producrion  •.'oiume  associated  v/itd  each  weapon  systerr.  locluaea 
Lr.  the  study.  Tvo  variaoies,  3EGTREND  and  ENDTREND,  '.vere  usea 
to  reflect  how  production  volume  was  changing  at  the  beginnina 
and  end  of  each  cost  series  to  which  models  were  fit. 
Production  rate  per  period  may  be  low  initially  in  order  to 
work  out  bugs  and  ensure  a  stable  product  design  prior  to  full 
scale  production.  .Alternatively,  initial  production  rate  per 
period  may  be  high  if  the  current  weapon  system  represents  an 
updated  version  of  an  already  existing  system  with  only  minor 
modifications . 

Production  rate  per  period  may  also  vary  at  the 
end  of  the  production  run  depending  on  whether  the  program  is 
abruptly  cancelled,  gradually  phased  out  or  continued  at  some 
minimum  level  in  order  to  maintain  a  warm  production  base. 
These  two  production  trend  variables  were  included  in  the 
study  to  determine  whether  the  bias  associated  v/ith  the 
various  models  was  related  to  past  production  trend. 
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The  production  trend  variables  were  measured  as  rollov/s: 


and 


END7RE1TD  =  - 

"1 

«  AV'G 


•.vhere 


3EGTREND  =  Production,  trend  at  the  beginning  or 
the  production  run. 

ENDTREND  =  Production  trend  for  the  most  recent  production 
period. 

=  Quantity  produced  in  first  production  period. 

Q^  =  Quantity  produced  in  most  recent  period, 

Qavg  ~  quantity  per  period  produced  through 

current  period. 


Positive  values  for  BEGTREND  and  ENDTREND  indicate  production 
trends  that  are  increasing  in  volume. 

(7)  Plot  Points 

The  final  independent  variable  included  in  the 
study  v;as  the  number  of  plot  points  (PLOTPNTS)  .  Plot  points 
indicates  the  number  of  data  points  available  to  estimate 
model  parameters.  The  accuracy  of  the  learning  curve, 
fixed-variable,  and  Bemis  models  should  improve  as  the  amount 
of  data  available  during  the  model  estimation  period 
increases.  Nevertheless,  if  a  model  is  inherently  biased. 
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r.or 


increasing  rhe  number  of  plot:  points  vill  not  necessar  i  _  v 
inmate  the  oias  associatea  v;ith  the  n.odei.  The  nuncer  :r 
clot  points  was  considerea  in  the  study  to  determine  -.'netner 
or  not  the  bias  associat  '.-/ith  the  various  models  could  te 
explained  in  terms  of  the  availability  of  past  production 
data . 


F .  SUMMARY 

This  chapter  has  provided  an  overview  of  the  sample  and 
variables  to  be  used  to  investigate  bias.  The  discussion 
began  with  a  description  of  the  sample  selection  process,  and 
a  summary  of  the  data  collected  for  each  program  included  in 
the  sample.  This  was  followed  by  a  description  of  the 
procedure  used  to  expand  the  sample  from  46  observations 
(programs)  to  121  cost  series.  Next,  the  four  cost  progress 
models  included  in  the  study- -the  random  walk  model,  the 
traditional  learning  curve  model,  the  fixed-variable  model, 
and  the  Bemis  production  rate  adjustment  model  were  described 
along  with  the  rationale  for  their  selection.  The  cost 
estimation  procedure  was  then  explained  and  a  method  for 
measuring  bias  was  introduced.  Finally,  two  categories  of 
independent  variables — demographic  variables  and  condition 
variables — were  introduced  to  be  evaluated  as  potential 
sources  of  model  bias. 
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IV.  ANALYSIS  AND  FINDINGS 

Chaprer  III  introduced  eleven  variables — four  demographic 
variables  and  seven  condition  variables — •■■/hich  might  influence 
the  performance  of  alternative  cost  progress  models  and, 
hence,  be  useful  in  explaining  the  cost  prediction  bias  of 
these  models.  This  chapter  describes  the  statistical 
procedures  that  were  performed  to  assess  the  significance  of 
each  of  these  variables  in  explaining  bias.  It  will  begin  by 
providing  an  overview  of  the  statistical  tests  that  were  used 
to  analyze  the  variables.  This  will  be  followed  by  a 
presentation  of  some  general  findings  with  respect  to  the 
performance  of  the  four  alternative  cost  progress  models. 
Next,  model  specific  findings  regarding  the  significance  of 
each  of  the  eleven  explanatory  variables  will  be  presented. 
Finally,  the  results  of  the  analysis  will  be  summarized  and 
conclusions  will  be  presented. 

A.  STATISTICAL  PROCEDURES 

Prior  to  conducting  statistical  analyses,  the  distribution 
of  values  for  each  of  the  dependent  and  independent  variables 
was  assessed.  The  presence  of  extreme  values  within  the  data 
set  could  unduly  influence  the  outcomes  of  the  statistical 
analyses.  Accordingly,  variable  values  which  lay  beyond  three 
standard  deviations  from  the  mean  for  the  variable  were 
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“runcated.  Iruncarion  involved  replacing  exoreme  values  ■..'itr. 
■.■a.Luas  equal  no  rhree  rines  rhe  srandara  aevianicn  rcr  me 
appropriate  variable.  This  approach  'vas  applied  because  :  r 
reduces  rhe  influence  of  ourliers  on  the  results  of  one 
statistical  analyses  without  discarding  and,  consequently, 
ignoring  the  impact  of  these  observations. 

Once  all  extreme  values  had  been  identified  and  truncated, 
the  dependency  of  BIAS  on  the  demographic  variables  was 
evaluated  for  each  model  using  analysis  of  variance  (ANOVA) . 
After  determining  the  significance  of  the  demographic 
variables  in  explaining  BIAS,  the  significance  of  the  seven 
condition  variables  and  MOD  was  assessed  using  both  simple  and 
multiple  linear  regression  analysis.  Finally,  three  different 
sets  of  Spearman  and  Pearson  correlation  analyses  were 
conducted.  First,  correlation  coefficients  were  computed  for 
the  measures  of  BIAS  from  the  four  different  models.  Next, 
correlations  between  the  condition  variables  (including  MOD) 
and  BIAS  were  examined  for  each  model.  Then  values  for  the 
condition  variables  (including  MOD)  were  correlated  with  each 
other  in  an  effort  to  detect  potential  muiticollinearity . 

In  conducting  tests  of  statistical  significance,  findings 
'.vith  alpha  values  less  than  0.01  were  considered  significant. 
When  analyzing  pairwise  correlations  between  the  explanatory 
variables,  correlation  coefficients  larger  than  0.50  were 
regarded  as  offering  strong  evidence  that  muiticollinearity 
might  be  a  problem. 
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B.  ANOVA  RESULTS 

Analysis  of  variance  -.vas  performed  for  eacn  of  r.he  four 
cost;  progress  models  to  determine  whether  or  not  prediction 
bias  could  be  explained  by  any  of  the  demographic  variables. 
The  following  demographic  variables  were  examined: 

1.  TYPE 

2.  HILSERV 

3.  MISSION 

4  .  MOD 

5.  TYPE  X  MOD  (interaction  variable) 

None  of  the  demographic  variables  tested  were  significant  in 
explaining  the  bias  of  cost  predictions  made  with  the  random 
walk,  fixed-variable  or  Berais  models.  However,  MISSION,  MOD 
and  the  TYPE  x  MOD  interaction  variable  were  all  found  to  be 
significant  in  their  ability  to  explain  variations  in  the  bias 
of  learning  curve  cost  predictions.  ANOVA  results  for  the 
learning  curve  model  are  provided  in  Table  4.  Findings 
concerning  the  significance  of  MOD  in  explaining  variations  in 
the  bias  of  learning  curve  cost  predictions  will  be  discussed 
later,  along  with  the  regression  results.  Differences  in 
learning  curve  bias  due  to  MISSION  and  TYPE  are  simply  noted. 
No  hypothesis  was  offered  to  expect  differences  in  learning 
curve  cost  prediction  bias  in  relation  to  MISSION  and  TYPE. 
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LEARNING  CURVE  MODEL  COST  PREDICTION  BIAS 
ANALYSIS  OF  VARIANCE  RESUl  TS 
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Consequently,  findings  related  to  these  factors  ■-■ere  not 
pursued  further  cut  are  nentioned  nere  for  possible 
investigation  in  future  researcn.  Based  on  the  fact  that  the 
utility  of  these  '/ariables  in  explaining  r.odel  bias  v.-as 
limited  to  the  learning  curve  model,  the  remainder  of  tne 
study  was  devoted  to  examining  the  significance  or  the 
condition  variables  (including  MOD)  in  explaining  model  bias. 

C,  OVERALL  MODEL  PERFORMANCE 
1.  Mean  Bias 

Summary  results  for  the  average  level  of  bias 
associated  with  cost  predictions  made  using  the  random  v/alk, 
learning  curve,  fixed-variable  and  Bemis  models  are  provided 
in  Table  5.  The  mean  bias  of  predicted  unit  costs  estimated 
with  the  four  models  ranged  from  -0.008313  for  the  learning 
curve  to  0.375045  for  the  fixed-variable  model.  The  results 
show  that  on  average,  the  learning  curve  underestimated  unit 
costs  by  approximately  0.33%  while  the  Bemis,  random  walk  and 
fixed-variable  models  overestimated  unit  costs  by 
approximately  2.9%,  4.6%  and  37.5%  respectively.  Bias 
measures  for  all  four  models  were  skewed  in  the  positive 
direction.  Hence,  measures  of  the  median  bias  '.vere  examined 
in  an  effort  to  obtain  a  more  objective  assessment  of  model 
performance . 
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RANK  ORDERING  OF 
OF  POSITIVE  3IAS 


MODELS  5Y  IIAGNITUDE 
;  Lowest:  zo  Hignesrl 


Model 

Mean  Bias 

Median 

Bias 

Standard 

Deviation 

Skewness 

Learning 

Curve 

-0 . 008313 

-0.060802 

0 . 335325 

: . 708457 

■ 

Bemis 

0.029472 

-0.012925 

0.339456 

0.338916  1 

Random 

Walk 

0.045842 

0.015789 

0.199953 

1 

1.396772 

Fixed- 

Variable 

0.375045 

0.188371 

0 . 929000 

3.391860  1 
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2.  Median  Bias 


The  nedian  bias  of  predicred  unit:  cosrs  esrir.aoed  v.-irr. 
the  four  models  ranged  from  -0.060802  for  the  learning  curve 
to  0.188371  for  the  fixed-variable  model.  When  the  median 
bias  measurements  for  the  various  models  were  compared,  the 
relative  ordering  of  the  models  according  to  magnitude  of 
positive  bias  remained  the  same.  However,  the  magnitude  of 
positive  bias  decreased  significantly  in  all  cases.  This 
suggests  that  some  observations  had  a  very  large  positive 
bias,  causing  the  mean  bias  to  be  more  positive  than  the 
median.  The  direction  of  bias  remained  unchanged  for  all 
models  except  for  the  Bemis  model.  In  this  case,  mean  model 
bias  was  approximately  2.9%  while  median  model  bias  was 
approximately  -1.3%.  These  results  seemed  to  indicate  that 
the  learning  curve  and  Bemis  models  underestimate  unit  cost 
(provide  low  unit  cost  estimates)  while  the  random  walk  model 
and  especially  the  fixed-variable  model  overestimate  unit  cost 
(provide  high  unit  cost  estimates) .  Perhaps  not 
coincidentally,  both  the  learning  curve  model  and  the  Bemis 
model  use  cumulative  quantity  to  predict  future  cost,  while 
neither  the  random  walk  model  nor  the  fixed-variable  model 
contain  a  cumulative  quantity  term. 
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3.  Model  Bias  Correlations 

3orh  Pearson  correlation  coetficients  ana  Jpearr.an 
correlation  coefficients  v.-ere  computed  for  the  -/a lues  or  cias 
associated  with  each  of  the  four  cost  progress  nodeis.  The 
results  of  these  correlation  analyses  are  provided  in  Table  . 
All  Spearman  correlation  coefficients  v;ere  positive  and 
significant;  correlation  values  ranged  from  0.27220  to 
0.58048.  However,  these  results  v/ere  not  completely  ccnfirmea 
by  the  computed  Pearson  correlicion  coefficients.  Values 
obtained  from  the  Pearson  correlation  analysis  i.ndicatea 
significant  bias  correlations  for  only  three  of  the  six  paired 
model  combinations:  BIASRW-BIASLC,  BIASRW-BIASBE ,  and  BIASBE- 
BIASFV.  Significant  Pearson  correlation  coefficients  ranged 
from  0.26401  to  .063335.  Both  correlation  analyses  indicated 
that  the  strongest  positive  correlation  existed  between  random 
walk  model  bias  and  learning  curve  model  bias.  The  second 
highest  correlation  in  both  analyses  existed  between  random 
walk  model  bias  and  Bemis  model  bias.  The  third  relationship 
•.vhich  was  correlated  and  significant  in  both  analyses  was 
Bemis  model  bias  and  fixed-variable  model  bias. 

Based  on  the  outcomes  from  the  correlation  analyses, 
it  was  concluded  that  there  is  evidence  of  a  positive 
correlation  between  bias  for  all  paired  model  combinations. 
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Table  6 


CORRELATION  ANALYSES  OF  MODEL  PREDICTION  BIAS 


Spearman  Correlation  Coefficients 


BIASRW 

BIASLC 

BIASFV 

BIASBE 

BIASRW 

1.00000 

0.58048* 

0.27220* 

0.346S8*  j 

o 

o 

0.0001 

0.0025 

0.0001 

BIASLC 

0.63335* 

1,00000 

0.35518* 

0.43751*  I 

0.0001 

0.0 

0.0001 

0 , 0001 

BIASFV 

-0.02073 

-0.04602 

1.00000 

0.54488* 

0.8215 

0.6162 

0.0 

0.0001 

BIASBE 

0.26401* 

0.13296 

0.35905* 

0.0034 

0.1460 

0.0001 

n 

Pearson  Correlation  Coefficients 


Correlation  Coe fficiencs/ Prob>  \r\  under  Rho=0 
*  Indicates  a  ^  0.01 
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rhar  being  zhe  case,  ic  seer.ea  reasonaole  zo  suszecz  chat,  co 
*:n2  extent  preaiction  cias  -.vas  pcsitivei'/  correiatea  a.-.ong  cne 
■/arious  .codeis,  the  ranao.ct  './aik,  learning  curce,  :ixea- 
variable  and  Semis  models  mignt  perform  in  a  sim.ilar  manner 
under  the  same  circumstances.  Moreover,  it  •.-.'as  possible  that 
the  significance  cf  various  explanatory  -/ariaD^ea 
explaining  prediction  bias  might  be  similar  between  models 
'.-.■hose  prediction  bias  was  highly  correlated.  In  order  to 
resolve  these  issues,  correlation  analyses  and  regression 
analyses  w-ere  used  to  study  the  relationship  between  m.odel 
bias  and  the  explanatory  variables.  The  following  paragraphs 
provide  the  results  of  these  analyses  for  each  of  the  four 
models . 

D.  REGRESSION  ANALYSES  AND  RESULTS 

The  significance  of  the  seven  condition  variables  and  one 
demographic  variable — MOD — was  assessed  using  regression 
analysis.  First,  simple  regression  analyses  were  performed  to 
independently  test  the  significance  of  these  independent 
variables  in  explaining  prediction  bias  when  considered  in 
isolation.  Then  multiple  regression  analyses  'were  conducted 
to  determine  the  significance  of  these  same  independent 
variables  in  explaining  prediction  bias  while  controlling  for 
the  effects  of  the  other  independent  variables.  Pairwise 
correlations  were  additionally  determined  among  both 
independent  and  dependent  variables.  The  follov/ing  sections 
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describe  the  results  of  these  analyses  for  each  of  the  four 
r.odels.  The  discussion  of  the  results  vill  be  organized  by 
model  rype  in  the  following  sections.  Summary  tables 
containing  average  bias  for  particular  subsets  of  the  sample 
(Table  7)  and  correlations  (Table  3)  are  provided  here.  They 
v/ill  be  referred  to  as  the  discussion  proceeds.  Regression 
results  will  be  presented  in  each  section  that  follows. 

1.  Random  Walk  Model  Bias  (BIASRW) 

Table  9  provides  the  results  of  the  multiple 
regression  analysis  for  BIASRW.  The  results  indicate  that 
approximately  33%  of  the  variation  in  random  walk  model  bias 
was  explained  by  the  nine  independent  variables.  However, 
only  two  variables — LRATE  and  FUTUPROD — were  significant  in 
explaining  variations  in  the  bias  of  random  walk  model  cost 
predictions.  The  relationships  between  random  walk  model  bias 
and  each  of  these  variables  are  depicted  in  Figure  1. 
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TABLE  8 


CORRELATION  AIJALYSIS  OF  INDEPENDENT 
VARIABLES  AND  MODEL  BIAS 


VARIABLES 


SPEARMAN  CORRELATION  COEFFICIENTS 
MODEL  BIAS 


. . 

BIASRW 

BIASLC 

BIASFV 

BIASBE 

PLOTPNTS 

-0.17974 

-0.05108 

0.10070 

MOD 

0.07265 

0.32932* 

0.00535 

0.15803 

BURDEN 

0.03855 

-0.15101 

-0.03751 

-0.08670 

CVAR 

0.09104 

-0.31316* 

0.13307 

-0.13021 

LRATE 

-0.07992 

0.34364* 

-0.25810* 

RATEVAR 

0 . 10188 

-0.06053 

0.02671 

-0.01795 

BEGTREND 

0.00121 

-0.28616* 

0.27543* 

ENDTREND 

0.13639 

-0.02809 

0.10935 

0.00670 

FUTUPROD 

0.42949* 

0.34583 

-0.28896* 

-0.11105 

PEARSON  CORRELATION  COEFFICIENTS 


VARIABLES  MODEL  BIAS 


BIASRW 

BIASLC 

BIASFV 

BIASBE 

PLOTPNTS 

-0. 18426 

-0.10901 

0.08183 

-0.04182 

MOD 

0.10132 

0.26970* 

-0.03685 

0 . 14173 

BURDEN 

0.21469 

0.29738* 

0.02674 

-0,07563 

CVAR 

0.03327 

-0.09775 

-0.02379 

-0.11590 

LRATE 

0.21866 

0.68339* 

-0.20121 

-0.01375 

RATEVAR 

0.04942 

-0.02878 

-0. 13466 

-0.12115 

BEGTREND 

-0,09221 

-0.35540* 

0. 13413 

0.02868 

ENDTREND 

0.08477 

0.07822 

0.03699 

-0.02651 

FUTUPROD 

0.46612* 

0.31543* 

-0.57893* 

-0 . 18493 

♦  Indicates  a  s  0.01 
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TABLE  9 


MULTIVARIATE 


Atrtl.  i  W  i. 


LANDOM  \vALK  MODEL  BIAS 


Analysis  of  Variance 


Sun  of  Mean 

Source 

DF 

Squares  Square 

F  Value 

Prob>F 

^■lodel 

9 

1.62871  0.18097 

6.709 

0.0001 

Error 

^6 

2.5S963  3.02698 

C  Total 

105 

-.21335 

Root  MSE 

0.16929  .R-square 

0.336  1 

Dep  Mean 

0.05092  Adj  R-sq 

0.3285 

c .  y . 

325.76890 

Parameter  Estimates 


Variable 

DF 

Parameter 

Estimate 

Standard 

Error 

T  for  HO: 
Parameter=0 

Prob  >  ITl 

INTERCEP 

1 

-0.366190 

0.15950058 

-2.296 

0.0239 

BURDEN 

1 

0.159379 

0.06505715 

2.950 

0.0161 

CVAR 

1 

-0.039291 

0.19791829 

-0.251 

0,8179 

IRATE 

1 

0.997789 

0.15651890 

2.861 

0.0052  * 

RATEVAR 

1 

-0.062376 

0.12325981 

-0.506 

0.6140 

BEGTREND 

1 

0.136197 

0.06080772 

2.239 

0.0275 

ENDTREND 

1 

0.099053 

0.02678170 

1 ,695 

0.1033 

FUTUPROD 

1 

0.126599 

0.02233301 

5.669 

0.0001  * 

PLOTPNTS 

1 

-0.016620 

0.00932812 

-1.782 

0.0780 

MOD 

1 

0 . 093861 

0 . 09639999 

1.059 

0.2999 

-  Indicates  a  5  0.01 
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RANDOM  WALK  BIAS 


0.25 


0.2 


0.15  i 


0.1 


0.05 


-0.05 


-0.1 


LRATE 


FUTUPROD 


■  low  □  medium  Uhigh 

Relationship  Between  Random  Walk  Model 
Bias  and  Significant  Explanatory  Variables 


Figure  1 
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a.  LRATE 


IRATE  '.vas  round  zo  ce  significant:  in  expiainir.cj 
variations  in  the  bias  of  random  walk  nodel  cost  predictions 
in  both  the  simple  and  multiple  regression  analyses.  The 
estimated  regression  coefficients  indicated  a  positive 
relationship  between  LRATE  and  BIASRW.  Computed  Spearman  and 
Pearson  correlations  were  inconclusive  with  regard  to  the 
nature  of  the  relationship.  Figure  1  indicates  that  the 
random  walk  model  bias  was  positive  for  all  levels  of  LRATE. 
However,  the  relationship  between  LRATE  and  3IASRVJ  appears 
counterintuitive.  One  would  expect  low  (high)  values  of  LRATE 
to  be  associated  with  high  (low)  values  of  bias.  In  fact,  a 
nonlinear  relationship  exists.  Bias  is  highest  (most 
positive)  when  the  learning  rate  is  at  extremes,  i.e.,  when 
the  learning  rate  is  steepest  or  most  shallow.  Bias  is  lowest 
when  the  learning  rate  is  in  the  middle  range,  i.e.,  where 
most  programs  likely  will  fall. 
t.  FUTUPROD 

In  addition  to  LRATE,  FUTUPROD  was  also  found  to  be 
significant  in  explaining  random  walk  bias  in  both  the  simple 
and  multiple  regression  analyses.  The  estimated  regression 
coefficient  was  positive  in  both  analyses  thus  indicating  a 
positive  relationship  between  FUTUPROD  and  BIASRW.  Computed 
Spearman  and  Pearson  correlation  coefficients  for  FUTUPROD  and 
BIASRW  were  both  moderately  positive  and  significant,  thereby 
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confirming  this  relationship.  The  results  show  rhat  when 
FUTUPROD  was  low  (high)  the  level  of  positive  bias  was  lev/ 
(high).  This  relationship  is  readily  apparent  in  Figure  1 
which  shows  clearly  that  when  FUTUPROD  was  low  (high)  ,  zhe 
random  walk  model  underestimated  (overestimated)  unit  costs. 
These  results  were  expected.  When  FUTUPROD  is  high  (low),  unit 
costs  are  normally  lower  (higher)  in  the  period  being  forecast 
because  of  decreasing  (increasing)  variable  costs  per  unit 
(learning  effect)  and  allocation  of  total  fixed  costs  over  a 
larger  production  volume. 

2.  Learning  Curve  Model  Bias  (BIASLC) 

Table  10  provides  the  results  of  the  multiple 
regression  analysis  for  BIASLC.  The  results  indicate  that 
approximately  73%  of  the  variation  in  learning  curve 
prediction  bias  was  explained  by  the  nine  independent 
variables.  Five  of  the  independent  variables  included  in  the 
multiple  regression  analysis  were  significant  in  explaining 
variations  in  the  level  of  bias  of  learning  curve  model  cost 
predictions.  These  variables  were:  BURDEN,  LRATE,  BEGTREND, 
ENDTREND,  and  FUTUPROD.  In  general,  the  simple  regression 
results  agreed  with  the  multiple  regression  results  in  terms 
of  the  significance  of  these  variables. 
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Analysis  of  Variance 


Cun  of  Mean 

Source  SF  Squares  Square 


Model 

9  12. 

^2818  1.43646 

r  -y 

.  G90  .0. 

r 

Error 

96 

Z  Total 

105  17. 

2254o 

Root 

MSE 

0 .21157 

R-sauare 

0  . 

7  c  n  d 

Dep 

Mea  n 

-0 . 00776 

Adj  R-Sq 

0  . 

>'  i.  .  i 

c.y . 

-2727 . 49365 

Paraneter  Estimates 

Parameter 

Standard  T 

for 

HO  : 

Variable 

DF 

Estimate 

Error  Pararieter  =  0 

Prob  >  IT! 

INTERCEP 

1 

-2.325371 

0.20546621 

-11 

.  318 

0 . 0001 

BURDEN 

1 

0.372917 

0.08380563 

4 

.450 

0  .0001 

CVAR 

1 

0  .  127763 

0 .19054602 

0 

.671 

0.504] 

IRATE 

1 

2.316325 

0.20162526 

11 

.  488 

0.0001 

* 

RATEVAR 

1 

-0.204656 

0 .15877497 

- 1 

.289 

0.2005 

DEGTREND 

1 

0.283523 

0.07833157 

J 

.620 

0  .  0005 

ENDTREND 

1 

0.175378 

0 . 03449978 

5 

.  083 

0  .0001 

FUTUPROD 

1 

0 .122591 

0 . 02876904 

4 

.261 

0  .  0001 

PLOTPNTS 

1 

-0 . 011255 

0 . 01201634 

-0 

.937 

0.3513 

MOD 

1 

0 .137628 

0 . 05970084 

-> 

.  305 

0 . 0  2  3  u 

However,  rhere  were  rwo  norev/orthy  differences:  1,  EMDTREHD 

..as  significant  -n  tne  -uitipie  regression  anaivsis  cut  v.as 
not  significant  in  the  sinpie  regression  analysis;  and  '2;  i'.OD 
was  significant  in  the  simple  regression  analysis  but  was  not 
significant  in  the  multiple  regression  analysis.  These  two 
findings  will  be  discussed  in  detail  later  in  this  section. 
The  following  paragraphs  discuss  the  results  of  the  regression 
analyses  for  each  of  the  variables  identified  as  being 
significant.  The  relationships  between  the  level  of  each  of 
the  significant  independent  variables  and  model  bias  are 
depicted  in  Figure  2. 
a.  BURDEN 

BURDEN  was  found  to  be  significant  in  explaining 
variations  in  model  bias  in  both  the  multiple  and  simple 
regression  analyses.  The  estimated  multiple  regression 
coefficient  indicates  there  was  a  positive  relationship 
between  burden  and  prediction  bias  for  the  learning  curve. 
This  result  was  confirmed  by  the  significant  but  relatively 
weak,  positive  Pearson  correlation  (See  Table  3)  .  These 
findings  indicate  that  when  the  proportion  of  total  cost  made 
up  of  fixed  costs  was  high  (low),  the  level  of  positive  bias 
was  high  (low).  Figure  2  graphically  depicts  the  relationship 
between  learning  curve  model  bias  and  BURDEN.  The  graph  shows 
that  low  (high)  positive  bias  was  indeed  associated  with 
low  (high)  levels  of  BURDEN. 
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BURDEN  URATE  BEGTREND  ENDTREND  FUTUPROD 

■  low  □  medium  Dhigh 

Relationship  Between  Learning  Curve  Model 
Bias  and  Significant  Explanatory  Variables 


Figure  2 


However,  the  graph  also  indicates  that  negative  cias  occurrc-o 
at  nedium  BURDEN  levels.  In  other  words,  when  t.he  propom cn 
of  total  cost  nade  up  of  fixed  costs  was  low,  the  learning 
curve  model  slightly  overestimated  unit  costs.  In  addition, 
when  the  proportion  of  total  cost  made  up  of  fixed  costs  v.’as 
at  a  medium  or  moderate  level,  the  learning  curve  model 
underestimated  unit  costs.  Finally,  when  the  proportion  of 
total  cost  made  up  of  fixed  costs  was  high,  the  learning  curve 
overestimated  unit  costs  by  a  moderately  large  amount.  This 
behavior  confirms  the  finding  in  the  earlier  study  by  Hoses 
[Ref.  5].  Moses  found  that: 

Negative  bias  consistently  increases  with  increases  in 
fixed  cost  burden~-up  to  a  point — then  negative  bias 
decreases  with  further  increases  in  burden.  The  turn 
around  point  for  all  observations  is  when  burden  is  50%. 
[Ref.  5:p.  27] 

Moses  attributes  this  behavior  to  the  fact  that  when  BURDEN  is 
0%  all  costs  are  variable  and  subject  to  learning.  In 
addition,  when  BURDEN  is  100%,  all  costs  are  fixed  and  are  not 
subject  to  learning.  Under  these  circumstances  the  learning 
curve  model  correctly  specifies  the  "true"  underlying  cost: 
function  and  no  bias  will  result.  According  to  Moses,  bias 
results  only  when  costs — some  subject  to  learning  and  some 
not — are  combined.  [Ref.  5:p.  28] 
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b.  LRATE 


LRATE  v.-as  rcund  co  ce  :5  igni  f  icanc  .n  cozh  ~r.c- 
sirr.pie  and  nultiple  regression  analyses.  Moreover,  IRATE  ..as 
the  most,  important:  variable  ..n  terms  ot  aoiiity  to  expiai.t 
'.'ariations  in  the  bias  of  learning  curve  model  predictions. 
This  is  evidenced  by  the  fact  that  its  t-value  far  exceedea 
the  t-values  for  the  other  explanatory  variables.  T.he 
positive  multiple  regression  coefficient  indicates  there  •..■as 
a  positive  relationship  between  LRATE  and  3IASLC.  This 
relationship  was  confirmed  by  the  computed  Spearman  ana 
Pearson  correlation  coefficients  (See  Table  8) .  The  Pearson 
correlation  coefficient  was  particularly  high  and  reflected  a 
relatively  strong,  positive,  linear  correlation  between  LRATE 
and  BIASLC.  The  results  indicate  that  when  LRATE  was  low 
(i.e.,  a  high  level  of  learning  was  occurring)  the  level  of 
positive  bias  was  low;  and,  when  LRATE  was  high  (i.e.,  a  low 
level  of  learning  was  occurring)  the  level  of  positive  bias 
was  high.  Figure  2  confirms  this  relationship.  When  the 
level  of  learning  was  high  (i.e.,  the  LRATE  was  low),  the 
learning  curve  model  greatly  underestimated  unit  costs.  V.-hen 
the  level  of  learning  was  moderately  high  (i.e.,  LRATE  was 
medium) ,  the  learning  curve  model  underestimated  unit  costs  by 
a  relatively  small  amount.  Finally,  when  the  level  of 
learning  was  low  (i.e.,  LRATE  was  high)  the  learning  curve 
model  greatly  overestimated  unit  costs. 
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The  findings  concerning  the  relationsnip  cetv/een 
IRATE  and  3IASLC  aescribed  aoove  differ  .'•.arkealy  :  rcn  cnc- 
results  presented  in  the  Moses  study  [Ref.  5i.  Moses  founa 
that  learning  rate  was  not  significant  in  its  ability  to 
explain  variations  in  learning  curve  bias.  However,  the 
results  of  the  current  study  indicate  that  learning  rate  is 
extremely  important  in  terms  of  its  ability  to  explain  .model 
bias.  This  suggests  that  the  traditional  learning  curve  model 
does  not  adequately  specify  the  affect  of  learning  on  variable 
costs . 

c.  BEGTREND 

BEGTREND  was  found  to  be  significant  in  both  the 
simple  and  multiple  regression  analyses.  Nevertheless,  the 
computed  regression  coefficients  provided  conflicting 
information  about  the  relationship  between  past  production 


trend  (measured  by 

BEGTREND) 

and 

BIASLC. 

The  simple 

regression  results 

(See  Table 

11) 

indicated 

a  negative 

relationship  while  the  multiple  regression  results  indicated 
a  positive  relationship. 

Computed  correlation  coefficients  for  BEGTREND  and 
BIASLC  were  examined  to  provide  another  look  at  the  nature  of 
the  relationship.  Both  Spearman  and  Pearson  correlation 
coefficients  reflected  the  existence  of  a  moderately  weak, 
negative  relationship  between  BEGTREND  and  BIASLC. 
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-ABLE  11 


UNIVARIATE  ANALYSIS  CF  LEARNING  CURVE  MODi 

EEGTREN'D) 


Analysis  of  Variance 


Source 

Model 
Error 
C  Total 


Root  MSE 
Dep  Mean 
C.7, 


Variable  DF 

INTERCEP  1 
3EGTREND  1 


Sun  of 
Squares 

3.55373 
.  3  .  o  b  6  7  3 
■7  .ZZ546 


3.55373 
0  .  1  -  4 1 


-  Value 


3.0001 


0 . 36251 
-0  .  00776 
-4673.24223 


S-sauare 
Ad3  R-sq 


0.1990 


Paraneter  Estinates 


Parameter 

Estimate 

0.191679 

-0.331017 


Standard  T  for  HO; 
Error  Paraneter=0 


0.05204281 

0.07321702 


3.683 

-5.204 


Prob  >  1 T I 

0.0004 

0.0001  » 
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This  indicates  that  '.-.•nen  the  past  production  trend  .,as 
decreasing  (increasing),  i.e.,  the  ^nitiai  proaucticn  ■.■ci-.r.e 
per  period  was  above  iceiow)  the  average  production  volume  per 
period),  the  level  of  positive  bias  was  high  ilowj.  The 
relationship  between  BEGTREND  and  BIASLC  is  depicted  in  Figure 
2.  The  graph  shows  that  when  the  past  production  trend  '..as 
decreasing  (i.e.,  BEGTREND  was  negative)  learning  curve  bias 
was  highly  positive.  In  addition,  when  past  production  trend 
w'as  relatively  stable  (i.e.,  BEGTREND  was  near  zero)  learning 
curve  bias  was  negative.  Finally,  the  graph  shows  t.hat  ■.■;hen 
the  past  production  level  was  increasing  (i.e.,  BEGTREND  v;as 
positive)  learning  curve  bias  was  highly  negative. 

In  general,  the  results  indicate  that  v;hen 
production  volume  at  the  beginning  of  a  series  of  production 
lots  (i.e.,  at  the  start  of  a  program)  starts  off  low  and 
subsequently  builds  upward  to  a  higher  volume,  the  learning 
curve  model  has  a  strong  tendency  to  underestimate  future 
costs.  In  contrast,  when  initial  production  volume  starts  off 
high,  the  learning  curve  is  biased  toward  overestimating 
future  costs. 

The  relationship  between  BEGTREND  and  BI.-.SLC 
described  above  is  consistent  with  the  results  of  Moses'  study 
in  terms  of  the  nature  of  the  relationship.  However,  Moses 
found  that  learning  curve  bias  was  negative  for  ail  production 
trends.  This  phenomenon  was  not  observed  in  the  current 
study. 
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In  an  etfor'c  lo  'iererr.ine  rhe  source  or  zr.c 
rcnrlicoinq  resuirs  iron  me  rearessicn  analyses,  ocrreiaricns 
cecween  3EGTREND  and  one  other  independGnt  -/'ar  rabies  ..-ere 
examined.  Correlations  oetween  the  independent  -/arrabies  are 
provided  in  Table  12.  The  correlations  indicate  significant, 
relatively  strong  relationships  existed  between  BEGTREND  ana 
CVAR,  BEGTREND  and  IRATE,  and  BEGTREND  and  RATEVAR. 
Consequently,  there  is  a  strong  possibility  that 
nulticollinearity  existed  between  these  variables.  Such  a 
condition  could  have  contributed  to  the  conflicting  regression 
results . 

Another  possible  factor  which  could  have  influenced 
the  results  is  that  there  may  have  been  interactions  between 
BEGTREND  and  the  other  variables  in  the  multiple  regression 
analysis.  Interactions  between  BURDEN  and  past  production 
trend  were  identified  in  the  Moses  study  and  were  found  to  be 
significant  in  explaining  variations  in  BIASLC. 
d.  ENDTREND 

ENDTREND  was  found  to  be  significant  in  explaining 
model  bias  in  the  multiple  regression  analysis  only.  This 
result  was  thought  to  be  largely  the  result  of  interactions 
between  ENDTREND  and  the  other  independent  variables.  The 
estimated  multiple  correlation  coefficient  indicated  a 
positive  relationship  existed  between  ENDTREND  and  BIASLC. 
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CORRELATION  ANALYSIS  OF  INDEPENDENT  VARIABLES 
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PEARSON  CORRELATION  COEFFICIENTS 


Hov/ever,  tihis  reiationship  could  not  be  confirned  cv  exairiininq 
tne  computed  correlation  coefficients.  iJeither  the  Pearson 
correlation  coefficient  not  the  Spearman  correlation 
coefficient  was  significant  for  the  relationship  between 
ENDTREND  and  BIASLC. 

The  positive  relationship  between  ENDTREND  and 
BIASLC  implies  that  when  the  past  production  trend  (ENDTREND) 
v/as  decreasing  (increasing)  (i.e.,  the  quantity  produced  in 
the  most  recent  period  was  less  than  (greater  than)  the 
average  quantity  produced  per  period) ,  the  level  of  positive 
bias  was  low  (high) .  This  relationship  between  ENDTREND  and 
BIASLC  is  shown  in  Figure  ?  The  graph  shows  that  when 
ENDTREND  was  low  (decreasing  production  trend)  BIASLC  was 
negative.  In  addition,  when  ENDTREND  was  medium  (relatively 
stable  production  trend)  BIASLC  was  less  negative.  Finally, 
the  graph  shows  that  when  ENDTREND  was  high  (increasing 
production  trend)  BIASLC  was  positive. 

In  the  most  general  terms,  the  results  indicate 
that  when  production  volume  at  the  end  of  a  series  of 
production  lots  is  declining,  the  learning  curve  model  has  a 
tendency  to  underestimate  future  costs.  Similarly,  when 
production  volume  at  the  end  of  a  series  of  production  lots  is 
increasing,  the  learning  curve  model  has  a  tendency  to 
overestimate  future  costs.  Note,  however,  these  tendencies 
are  mild,  as  indicated  by  the  small  magnitude  of  the  effects 
in  Figure  2. 
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e.  FUTUPROD 


FUTUPROD  v;as  found  to  be  significant  in  expiaininq 
nodel  bias  in  both  the  simple  and  multiple  regression 
analyses.  The  estimated  regression  coefficient  v/as  positive 
in  both  analyses  thus  indicating  a  positive  relationship 
between  FUTUPROD  and  3IASLC.  Computed  Spearman  and  Pearson 
correlation  coefficients  for  FUTUPROD  and  BIASLC  were  both 
•.veakly  positive  and  significant,  thereby  confirming  this 
relationship.  The  results  indicate  that  when  FUTUPROD  was 
loiv  (high)  the  level  of  bias  was  low  (high)  .  This 
relationship  is  readily  apparent  in  Figure  2.  The  graph  shows 
that  when  FUTUPROD  was  low,  BIASLC  was  highly  negative.  When 
FUTUPROD  was  medium  (i.e.,  the  production  level  for  the  next 
period  was  comparable  with  the  production  level  for  the  last 
period)  BIASLC  was  moderately  negative.  Finally,  when 
FUTUPROD  was  high,  BIASLC  was  highly  positive. 

The  relationship  between  FUTUPROD  and  BIASLC 
observed  in  this  study  coincides  almost  exactly  with  the 
relationship  described  in  the  Moses  study  [Ref.  5].  As 
expressed  by  Moses,  this  relationship  sh'^uld  be  expected. 

Higher  (lower)  future  production  will  result  in  lower 
(higher)  fixed  cost,  and  total  cost,  per  unit,  creating  a 
tendency  toward  positive  (negative)  bias  for  any  cost 
estimate.  [Ref.  5:p.  22] 

In  general,  these  findings  indicate  that  the 
learning  curve  model  tends  to  underestimate  unit  costs  when  it 
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is  used  to  predict  costs  for  periods  ir  wnicn  producricr. 
'.olurie  is  cut  cack.  icr.verseiV,  the  ieamr.q  lurve  -coe_ 
rends  to  overest  imare  uniz  costs  for  perioas  in  •..'nicn 
production  volume  is  increased. 
f.  MOD 

MOD  was  identified  as  being  significant  in 
explaining  variations  in  bias  in  the  simple  regression 
analysis  (See  Table  13);  however,  MOD  was  not  significant  in 
the  multiple  regression  analysis.  Both  Spearman  and  Pearson 
correlation  coefficients  were  examined.  The  correlation 
coefficients  indicated  a  significant,  weakly  positive 
correlation  between  MOD  and  BIASLC.  This  suggests  that  the 
learning  curve  model  has  a  lesser  tendency  to  underestimate 
unit  costs  when  used  in  predicting  costs  for  modification  type 
programs.  Examination  of  the  correlations  between  MOD  and  the 
other  explanatory  variables  revealed  there  was  a  significant, 
moderately  strong  positive  correlation  between  MOD  and  LRATE. 
Consequently,  it  is  likely  that  multicollinearity  betwee.n  MOD 
and  LRATE  was  the  source  of  the  lack  of  significance  for  MOD 
in  the  multiple  regression  analysis.  This  positive 

correlation  between  MOD  and  LRATE  seems  reasonable.  One  'would 
expect  modification  programs  to  exhibit  less  learning  than 
programs  involving  entirely  new  designs. 
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TABLE  13 


UNIVARIATE  ANALYSIS  OF  LEARNING  CUI.VE  MODEL  BIAS 

:M0D) 


Analysis  of  Variance 


Source 

DF 

Sum  of 
Squares 

Mean 

S  qua  re 

F  Value 

Model 

1 

1.19852 

1.19852 

7 .777 

Error 

1  nc 

16  .  02694 

0.15411 

C  Total 

105 

17 . 22546 

Root  MSE  0.39256  R-souare 
Dep  Mean  -0.00776  Adj  R-sq 
C.y .  -5060.70355 


0. 0696 
0.0606 


Parameter  Estimates 


Variable  DF 

INTERCEP  1 
MOD  1 


Parameter 

Estimate 

-0.156032 

0.224530 


Standard 

Error 

0.06542706 

0.03051213 


T  for  HO ! 
Parameter=0 

-2.385 

2.739 


Prob  >  |T| 

0.0189 
0 .0063  * 


*  Indicates  ct  ^  0.01 


38 


3.  Fixed-variable  Model  Bias  (BIASFV) 

Table  14  provides  zhe  results  :t  z'r.e  ruitio,- 
repression  analysis  ror  3IASFV.  The  results  indicatea  'tar 
approximately  37%  of  the  variation  in  fixed-variable  ncae^ 
oias  was  explained  oy  the  nine  independent  •/ariaDles. 
Hov/ever,  only  one  of  these  variables  —  r  UTUPROD-- was 
significant  in  explaining  variations  *n  the  bias  of  fixea- 
variable  model  cost  predictions.  FUTUPROD  was  also  found  to 
be  significant  in  the  simple  regression  analysis.  The 
estimated  multiple  regression  coefficient  indicates  a  negative 
relationship  existed  between  FUTUPROD  and  BIASFV.  This 
relationship  is  confirmed  by  the  computed  Spearman  and  Pearson 
correlation  coefficients  for  FUTUPROD  and  BIASFV.  Both 
correlation  coefficients  indicate  the  existence  of  a 
significant  negative  correlation.  Moreover,  the  relatively 
large  negative  Pearson  correlation  coefficient  indicates  that 
the  correlation  between  FUTUPROD  and  BIASFV  was  fairly  linear. 
The  relationship  between  fixed-variable  model  bias  and 
FUTUPROD  is  depicted  in  Figure  3. 

The  graph  confirms  that  when  the  level  of  production 
in  the  next  period  was  high  (low)  relative  to  the  most  recent 
period,  the  level  of  positive  bias  was  low  (high).  In  short, 
the  fixed-variable  model  has  a  tendency  to  overestimate  future 
costs.  This  tendency  is  greatest  when  the  model  is  used  to 
predict  costs  for  periods  in  which  cutbacks  in  production 
volume  occur. 
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lABLE  14 


MULTIVARIATE  ANALYSIS  OF 


:XED-VARIABLE  MODEL 


Analysis  of  Variance 


Sun  of  Mean 

Source 

DF 

Squares  Square 

F  Value 

Model 

9 

92.65513  9.73946 

7  .  397 

Error 

96 

57.98986  0.60901 

C  Total 

105 

100.63998 

Root  MSE 

0.77718  R-square 

0 . 9238 

Dep  Mean 

0.92697  Adj  R-sq 

0 . 3698 

C.  V. 

182.23765 

Parameter  Estimates 


Variable 

DF 

Parameter 

Estimate 

Standard 

Error 

T  for  HO: 
Parameter=0 

Prob  >  ITl 

IHTERCEP 

1 

1.627186 

0.75979657 

2.156 

0 . 0336 

BURDEN 

1 

0.351109 

0.30789621 

1.191 

0 . 2569 

CVAR 

1 

-0.833763 

0.69993999 

-1 . 191 

0 . 2365 

IRATE 

1 

-1.572097 

0.79063728 

-2.123 

0 . 0369 

RATEVAR 

1 

-0.016999 

0.53323376 

-0 . 029 

0.9769 

BEGTREND 

1 

-0.229979 

0.28773817 

-0.782 

0.9362 

ENDTREND 

1 

0.010113 

0.12672928 

0.080 

0 . 9366 

FUTUPROD 

1 

-0.751819 

0.10567336 

-7 .119 

0.0001  » 

PLOTPNTS 

1 

0  .  099066 

0 . 09919006 

1.112 

0.2691 

MOD 

1 

0 . 099669 

0.21930129 

0.952 

0.6670 

*  Indicates  a  <  Q.Ol 
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FIXED-VARIABLE  BIAS 


1 


FUTUPROD 


■  low  L  medium  Uhigh 

Relationship  Between  Fixed-Variable  Model 
Bias  and  Significant  Explanatory  Variable 

Figure  3 
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4.  Bemis  Rate  Adjustment  Model  Bias  (BIASBE) 

Table  15  provides  “be  results  sf  ~ne 
regression  analysis  ror  BIASBE.  Only  approximately  n  z:'e 
'-■ariation  in  Bemis  model  prediction  bias  was  explained  cy  tne 
nine  independent  variables.  Moreover,  none  of  the  variables 
■.v'ere  found  to  be  significant  in  either  the  simple  regression 
or  multiple  regression  analyses.  This  means  that  the  Bemis 
model  was  successful  in  accounting  for  the  influences  of  t.hese 
variables.  The  findings  concerning  Bemis  model  performance 
support  the  conclusions  drawn  in  the  earlier  Moses  stuay. 
Moses  found  that  the  overall  mean  bias  for  ail  cost 
predictions  made  with  the  Bemis  rate  adjustment  model  was 
-0.0016.  As  a  result,  he  concluded  that,  on  average,  the  rate 
adjustment  model  exhibits  no  bias.  In  addition,  Moses 
observed  that  the  absence  of  bias  was  evident  for  all 
treatments  across  all  variables  of  interest.  No  significant 
main  effects  were  observed  in  the  ANOVA  results.  [Ref.  5:p.24[ 


92 


TABLE  15 


MULTIVARIATE  AMALVSIS  OF  :BEMIS  MODEL 


Source 

DF 

Analysis  of  Variance 

Sum  of  Mean 

Squares  Square 

F  Value 

Prab>F 

Model 

9 

1.35915  0.14379 

1.194 

0.3081 

Error 

C  Total 

96 

105 

11.96836  0,12467 

13.30751 

Root 

MSE 

0 . 35309 

R-squa  re 

0.1006 

Dep 

Mean 

0 . 03091 

Adi  R-sq 

0 . 0165 

C.V. 

1142.35596 

Parameter  Estimates 

Parameter 

Standard 

T  for  HO: 

Variable 

DF 

Estimate 

Error 

Parameter^  0 

Prob  >  ITI 

INTERCEP 

1 

0.097655 

0.34289489 

0.285 

0.7764 

BURDEN 

1 

-0.055001 

0 .13986008 

-0.393 

0.6950 

CVAR 

1 

0.161554 

0.31799513 

0.508 

0,6126 

IRATE 

1 

-0 . 040905 

0 . 53648487 

-0.122 

0.9035 

RATEVAR 

1 

-0.424442 

0.26497361 

-1.602 

0.1125 

BEGTREND 

1 

0.090834 

0.13072464 

0.695 

0 . 4838 

ENDTREND 

1 

0 , 044540 

0.05757540 

0.774 

0.4411 

FUTUPROD 

1 

-0 . 075044 

0.04801158 

-1 ,563 

0.1213 

PLOTPNTS 

1 

-0 . 008266 

0.02005362 

-0.412 

0.6811 

MOD 

1 

0  .  192540 

0.09963251 

1  .933 

0.0562 
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SUMMARY 


"his  chapter  cas  iescribea  che  results  ctarist^ra. 
analyses  of  the  bias  associated  Vvith  unit  cost  preaicticnr 
obtained  using  the  random  walk,  traditional  learning  curve, 
f ixed- variable  and  Semis  oroduction  rate  adjustment  models. 
The  discussion  began  with  a  description  of  the  statistical 
procedures  used  in  conducting  the  study.  This  was  followed  by 
a  presentation  of  the  findings  with  respect  to  overall 
performance  of  the  four  cost  progress  .models  and  in 
examination  of  t.he  factors  thought  to  be  useful  in  expia..nina 
variations  in  model  performance  (bias) . 

Median  bias  values  for  the  four  models  indicated  t.hat  the 
traditional  learning  curve  model  and  the  Bemis  model  tend  to 
underestimate  the  unit  costs  of  weapons  systems  while  the 
random  walk  and  fixed-variable  models  tend  to  overestimate 
unit  costs.  In  addition,  correlation  analyses  of  the  cost 
prediction  bias  associated  with  each  of  the  models  reflected 
a  significant  positive  correlation  between  predicted  unit  ccsr 
bias  for  all  paired  model  combinations.  ANOVA  and  regression 
analyses  were  conducted  to  determine  the  significance  of  the 
four  demographic  variables  and  eight  condition  variables 
included  in  the  study.  Only  three  of  the  demographic- 
variables — MISSION,  MOD  and  TYPE  x  MOD  (interaction 
variable) — were  significant  in  terms  of  their  ability  to 
explain  variations  in  the  level  of  model  prediction  bias. 
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Moreover,  the  sianirioance  of  rhese  variaoies  '-.'as  linirea  to 
t.te  .earning  curve  .-.oaei. 

Findings  concerning  one  significance  of  the  ‘i^ant 
condition  variables  reveaiea  that  the  ctilitv  of  these 
variables  in  explaining  node!  cost  prediction  bias  variea 
widely  among  the  models,  only  two  condition  variabies--LRATE 
and  FUTUPROD — were  significant  in  explaining  variations  in  the 
bias  of  random  walk  cost  predictions.  Conversely,  five 
condition  variables — BURDEN,  LPvATE,  BEGTREND,  ENDTREND  and 
FUTUPROD — were  significant  in  explaini.ng  variations  in  the 
bias  of  learning  curve  model  cost  predictions.  FUTUPROD  was 
the  only  condition  variable  that  was  significant  in  explaining 
variations  in  the  bias  of  fixed-variable  model  cost 
predictions.  Finally,  none  of  the  condition  variables  was 
significant  in  explaining  variations  in  the  bias  of  Berais 
model  cost  predictions. 

The  results  of  the  statistical  analyses  of  learning  curve 
cost  prediction  bias  differed  somewhat  from  those  obtained  by 
Moses  [Ref.  5],  particularly  with  respect  to  the  significance 
of  LRATE  and  PLOTPNTS  in  explaining  variations  in  cost 
prediction  bias.  However,  results  for  the  Bemis  production 
rate  adjustment  model  strongly  supported  the  findings 
presented  by  Moses.  Chapter  V  will  summarize  the  major 
findings  from  the  current  study  and  will  suggest  directions 
for  future  research. 
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V. 


SUMMARY 


A.  FINDINGS 

The  primary  objective  of  this  study  v;as  to  determine  rr.e 
bias  of  selected  cost  progress  models  when  predicting  the 
future  unit  cost  of  weapons  systems  acquired  through  a 
continuing  acquisition  program.  In  addition,  the  research 
sought  to  answer  the  following  questions: 

1.  Are  the  various  cost  progress  models  comparaole  in  terms 
of  bias? 

2.  Do  particular  models  result  in  less  biased  estimates 
under  certain  conditions? 

3.  What  are  those  conditions  that  affect  the  performance  of 
the  models? 

4.  Can  guidelines  be  established  for  determining  when  (under 
what  conditions  or  circumstances)  it  is  most  appropriate 
to  use  a  particular  model  type? 

This  chapter  will  address  each  of  these  areas  by  summarizing 
the  results  of  the  study. 

Overall  findings  with  respect  to  the  bias  of  unit  cost 
predictions  obtained  with  the  random  walk,  traditional 
learning  curve,  f ixed-variable,  and  Bemis  production  rate 
adjustment  models  were  as  follows: 

1.  On  average,  the  learning  curve  model  underestimated  unit 
costs  by  approximately  6.1%. 
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2.  On  average,  the  Bemis  nodel  underestimated  unit  cost;:  cy 
approx inateiy  1.2%. 

3.  On  average,  the  random  walk  model  overest imatea  unit 
costs  by  1.6%, 

4.  On  average,  the  fixed-variable  model  overestimated  unit 
costs  by  13.3%. 


These  findings  indicate  that  the  four  models  do  indeed  differ 
in  terms  of  the  direction  and  magnitude  of  cost  prediction 
bias . 

The  influence  of  various  factors  on  model  performance  was 
examined  by  evaluating  the  significance  of  four  demographic 
variables  and  eight  condition  variables  in  explaining 
variations  in  the  bias  of  unit  cost  predictions.  Findings 
were  that  the  utility  of  these  variables  in  explaining  model 
cost  prediction  bias  varied  widely  among  the  models.  In 
particular,  the  following  relationships  were  observed: 


1.  Random  walk  model  cost  prediction  bias  is  influenced  by 
two  factors — the  learning  rate  associated  with  the 
production  process,  and  the  production  level  for  the 
future  period  relative  to  the  most  recent  period. 

2.  When  the  level  of  learning  is  high,  the  random  walk  model 
overestimates  unit  costs  by  approximately  5.2%.  At 
moderately  high  learning  levels,  the  random  walk  model 
overestimates  unit  costs  by  only  approximately  2.3%. 
However,  when  the  level  of  learning  is  low,  the  randomi 
walk  model  overestimates  unit  cost  by  approximately  7.5%, 

3.  The  higher  the  production  level  in  a  future  period  (the 
period  for  which  unit  costs  are  being  forecast)  relative 
to  the  most  recent  production  period,  the  more  the 
random  walk  model  tends  to  overestimate  future  cost. 
When  the  future  production  level  is  significantly  lower 
than  the  level  in  the  most  recent  production  period,  unit 
costs  are  underestimated  by  approximately  6.4%.  When  the 
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future  production  level  is  comparaole  vith  “.r.e 
recent:  perioa's  prcducncn  level,  unit:  uosrs  irc 

overest inarea  by  approxica-ceiy  Z.iu.  Finail,y,  /nen  me 
future  production  level  is  significant:!'/  nigner  than  me 
level  in  the  tost  recent  production  period,  unit  ;csm 
are  overestimated  by  approximately  20.1%. 

4.  Learning  curve  model  cost  prediction  bias  is  inrluencea 
by  the  following  factors:  the  percentage  of  total  ucst 
.Tiade  up  of  fixed  costs,  the  level  of  learning  associatca 
with  the  production  process,  the  past  production  trend, 
and  the  future  production  level. 

5.  When  the  proportion  of  total  cost  made  up  of  fixed  costs 
varies,  learning  curve  mciel  bias  is  affected  as  follows, 
.^it  low  levels,  unit  costs  are  overestimated  by 
approximately  2,5%;  at  medium  levels,  unit  costs  are 
underestimated  by  approximately  8.3%;  at  high  levels  unit 
costs  are  overestimated  by  approximately  11.1%. 

6.  The  higher  the  level  of  learning,  the  more  positive  the 

bias  of  learning  curve  model  unit  cost  estimates.  When 
the  level  of  learning  is  high,  costs  are  underestimated 
by  approximately  15.2%.  When  the  level  of  learning  is 
moderately  high,  costs  are  underestimated  by 

approximately  5,4%.  When  the  level  of  learning  is  low, 
costs  are  overestimated  by  approximately  22.0%. 

7.  When  the  production  volume  per  period  is  decreasing  at 
the  beginning  of  a  series  of  production  lots,  the 
learning  curve  model  overestimates  unit  costs  by 
approximately  19.5%.  When  the  beginning  production  trend 
is  relatively  stable,  the  learning  curve  model 
underestimates  unit  costs  by  approximately  5.6%.  when 
the  production  volume  per  period  at  the  beginning  of  a 
series  of  production  lots  is  increasing,  the  learning 
curve  model  underestimates  unit  costs  by  approximately 
12.2%. 

8.  When  the  production  volume  per  period  is  decreasing  at 
the  end  of  a  series  of  production  lots,  the  learning 
curve  model  underestimates  unit  costs  by  approximately 
3.9%.  When  the  ending  production  trend  is  relatively 
stable,  the  learning  curve  model  underestimates  unit 
costs  by  approximately  1.4%.  When  the  production  volume 
per  period  is  increasing  at  the  end  of  a  series  of 
production  lots,  the  learning  curve  model  vvill 
overestimate  unit  costs  by  approximately  3.2%. 
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9.  There  is  a  positive  relationship  between  the  level  of 
production  in  a  future  period  (relative  to  the  tost 
recent  period)  and  learning  curve  model  cost  predicticn 
bias.  When  the  level  of  production  in  a  future  period  is 
significantly  lower  than  the  level  of  production  in  the 
most  recent  period,  the  learning  curve  mcdei 
underestimates  unit  costs  by  approximately  17.0%.  When 
the  future  production  level  is  comparable  with  the 
production  level  in  the  most  recent  period,  the  learning 
curve  model  underestimates  unit  costs  by  approximately 
4.9%.  Finally,  when  the  future  production  level  is 
significantly  higher  than  the  level  of  production  in  the 
most  recent  period,  the  learning  curve  model  will 
overestimate  unit  costs  by  approximately  23.6%. 

10.  Fixed-variable  model  cost  prediction  bias  is  influenced 
by  the  level  of  production  in  a  future  period.  There  is 
a  negative  relationship  between  the  level  of  production 
in  a  future  period  (relative  to  the  most  recent  period) 
and  fixed-variable  model  bias.  When  the  future 
production  level  is  significantly  lower,  the  fixed- 
variable  model  overestimates  unit  costs  by  approximately 
9.1%.  When  the  future  production  level  is  relatively 
stable,  the  fixed-variable  model  overestimates  unit  costs 
by  approximately  26.8%.  Finally,  when  the  future 
production  level  is  significantly  higher,  the  fixed- 
variable  model  overestimates  unit  costs  by  approximately 
6.0%. 

11.  Bemis  production  rate  adjustment  model  cost  prediction 
bias  is  not  significantly  related  to  any  of  the  variables 
included  in  the  study. 


The  selection  of  a  particular  cost  progress  model  for 
estimating  airframe  unit  costs  depends  primarily  on  the 
availability  of  requ(.red  data  and  the  cost  versus  the  benefits 
of  collecting  tf  a  additional  data  required  to  employ  more 
sophisticated  cost  prediction  models.  The  findings  from  this 
study  indicate  that  the  learning  curve,  Bemis  and  random  walk 
models  all  produce  cost  predictions  which  have  low  biases. 
However,  the  models  differ  widely  in  terms  of  their 
susceptibility  to  vagaries  in  the  production  process.  Ceteris 
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paribus,  the  Berais  todel  is  superior  to  the  tther  :c;tt 
progress  models  because  it  not  only  txnibits  the  smallest  oias 
cut  also  IS  not  significantly  influenced  ^ in  terms  of  bias;  cy 
variations  in  the  factors  considered  in  this  study.  Hence,  :t 
provides  the  most  robust  and  consistent  cost  estimates. 
Conversely,  the  bias  of  unit  cost  predictions  obtained  ■vith 
the  random  walk,  learning  cur'/e,  and  f ixed-variable  models  is 
significantly  influenced  by  variations  in  the  production 
process.  Consequently,  the  findings  presented  above  should  be 
considered  when  employing  these  less  sophisticated  models. 


B.  FUTURE  RESEARCH  OPPORTUNITIES 

The  results  obtained  in  the  study  suggest  other  potential 
directions  for  future  research: 


1.  The  current  study  could  be  extended  to  include  aircraft 
engines  and  missile  propulsion  systems  to  determine 
whether  the  findings  concerning  model  prediction  bias 
hold  when  the  models  are  used  to  predict  unit  costs  for 
these  systems, 

2 .  Further  studies  could  be  conducted  to  determine  the  exact 
nature  of  the  relationships  between  aircraft 
mission/missile  mission,  modification  status,  type  of 
system  {aircraft  or  missile)  and  learning  curve  model 
prediction  bias. 
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